BioRevolution #20 - DNA Language Model (DLM)
We talk about letters that can kill you or cure you.
This text is written in English. It is translated into other languages thanks to my colleagues. We are communicating. I have some knowledge, and I am sharing it with you. You receive this information, and depending on your interest, you may generate an idea, take an action, or simply enjoy learning something new.
A similar process takes place inside our cells.
DNA stores information. This information is transcribed into messenger RNA. The ribosome then reads this message and produces proteins as the final output. Information flows from storage to interpretation to action.
For that reason, DNA can be viewed as a language. It has an alphabet, structure, and rules that determine how information is interpreted.
If I make a “mistkae” while writing this text, the consequence is limited. In biology, the stakes are different. A single change in DNA can alter how a protein is produced, sometimes with serious consequences for the organism. So, we are talking about language’s letters that can kill you or cure you.
In our daily lives, we interact with systems such as ChatGPT, Gemini, and Claude. These are examples of Large Language Models (LLM). They process human language as input, convert it into mathematical representations, and generate an output.
A DNA language model follows a similar idea, but the language is different.
Instead of words, the system works with sequences composed of four nucleotides: A, T, G, and C. These sequences carry biological information, but interpreting them is far more complex than reading a sentence. The meaning of a sequence depends on context, structure, and interactions within the cell.
The genome contains both coding and non-coding regions. Coding regions provide instructions for building proteins. Non-coding regions do not directly produce proteins, but they regulate when, where, and how genes are expressed. (They account for 98% of the genome.) Although they make up the majority of the genome, their function is still not fully understood.
Understanding this regulatory layer is one of the central challenges in modern biology.
For a long time, decoding DNA at scale was beyond our capabilities. Today, advances in machine learning are changing that. DNA language models are emerging as a new way to interpret genomic information, bringing us closer to understanding how this biological language operates.
Welcome to the DNA Language Model (DLM) world!
BioRevolution #20 - DNA Language Model (DLM) issue is supported by Capital Cell, which is Europe’s leading investment platform specialized in biotech and health startups, connecting groundbreaking science with forward-thinking investors. This article is available in properly written 🇫🇷 French and 🇪🇸 Spanish on the Capital Cell blog.
Scientific Perspective
A DNA language model is a machine learning system trained on large collections of genomic sequences. Instead of learning from human text, it learns from DNA.
In this setting, the genome can be viewed as a form of biological text. The basic units are four nucleotides: adenine (A), thymine (T), cytosine (C), and guanine (G). Different combinations of these letters form patterns that carry biological meaning. The objective of a DNA language model is to learn these patterns.
https://theory.labster.com/de/dna-structure/
Unlike traditional models in biology, which rely on labeled experimental data, DNA language models are trained using a self-supervised approach. This means the model learns directly from raw DNA sequences without requiring explicit annotations. By analyzing billions of nucleotides across different organisms, the system builds an internal representation of how genomic information is structured. This process allows the model to capture multiple layers of information at once. At a local level, it can recognize short patterns such as transcription factor binding sites. At a broader scale, it can begin to understand how distant regions of the genome interact, including regulatory elements that may be separated by thousands of base pairs.
This shift is important. Instead of training a new model for every biological question, a single pretrained system can be adapted to many tasks, such as predicting the effect of genetic mutations, modeling gene regulation, or designing new sequences.
How DNA Language Models Work
Despite their complexity, most DNA language models follow a similar structure. The system can be understood as three main components: how the sequence is represented, how patterns are learned, and how predictions are generated.
Learning Without Labels
The first step is pretraining. Most DNA language models are trained using one of two strategies.
In the first approach, parts of the DNA sequence are hidden, and the model is asked to predict the missing pieces. To do this correctly, the system must understand the surrounding context. This method allows the model to learn relationships between nearby and distant regions of the genome.
In the second approach, the model is trained to predict the next nucleotide in a sequence based on what came before. This setup is similar to how text generation models work. It is particularly useful when the goal is to generate new DNA sequences or complete incomplete ones.
Both methods force the model to learn the underlying structure of DNA rather than memorizing specific examples.
Representing DNA as Tokens
Before a model can learn from DNA, the sequence must be converted into units that the system can process. This step is called tokenization.
In human language, words are separated by spaces. DNA does not have natural boundaries. It is a continuous sequence of letters. This makes tokenization more challenging.
A common approach is to group nucleotides into short segments of fixed length. For example, a sequence of six letters can be treated as a single unit. This helps the model capture local patterns, but it can introduce redundancy and increase computational cost.
More advanced approaches use adaptive methods that group frequently occurring patterns into larger units while keeping rare sequences more detailed. This allows the model to represent the genome more efficiently while preserving important biological signals. This design choice affects how well the model can detect meaningful structures in DNA, especially in regions that regulate gene activity.
a, The principle of BPE highlighted on an example sequence with the tokenization steps relevant for this sequence. Resulting vocabularies are coloured by token length and depicted in a word cloud with relative weights of the words by their frequency. b, The model is a BERT architecture with 12 transformer blocks (in purple), which use multihead attention and a feed forward layer with normalisation steps (LayerNorm) in between. The model is embedding the tokens and is trained with cross-entropy loss to predict the masked token and updates the embedding while training. The output is probabilities of the masked token identity. https://www.nature.com/articles/s42256-024-00872-0/figures/1
Neural Architectures and Long Sequences
The next challenge is scale. Genomic sequences are long. Important interactions can occur across tens of thousands of nucleotides. Standard machine learning models struggle to handle this length efficiently. Many DNA language models are based on transformer architectures, similar to those used in natural language processing. These models are effective at capturing relationships within sequences, but their computational cost increases rapidly with sequence length.
To address this limitation, newer architectures are being developed. Some models replace attention mechanisms with more efficient operations that can process longer sequences. Others introduce memory components or alternative mathematical frameworks that scale more effectively. These innovations allow models to analyze larger regions of the genome at once, which is critical for understanding regulatory mechanisms that are distributed across distant locations.
A Different Kind of Language
Although the analogy to human language is useful, DNA operates under different rules.
There are no clear words or sentences. The same sequence can have different meanings depending on its context. In addition, biological processes such as transcription introduce directionality that is not always obvious from the raw sequence alone. Most importantly, a large portion of the genome does not directly encode proteins. Instead, it regulates when and how genes are expressed. These regulatory regions are complex and remain only partially understood.
For this reason, learning the “language of DNA” is not simply about reading sequences. It requires understanding how information is organized, interpreted, and executed within a living system.
Market Perspective
Global Genome Language Modeling (GLM) Market Size is valued at US$ 31.2 Bn in 2024 and is predicted to reach US$ 101.6 Bn by the year 2034 at an 12.9% CAGR during the forecast period for 2025-2034. (Genomic Language Models (gLMs) and DNA Language Models (DNA LMs) are often used interchangeably to describe AI models trained on DNA/RNA sequences to predict functional elements.)
Source: https://www.insightaceanalytic.com/report/genome-language-modeling-market/3232
Tools and Platforms
For researchers and developers, several entry points exist for utilizing DNA language models.
Open-Source Repositories and Libraries
HuggingFace has become the central repository for pretrained DNA models, including the Nucleotide Transformer, DNABERT-2, and GENA-LM. These models can be loaded into standard Python-based deep learning workflows using the transformers library. Additionally, several specialized toolkits exist on GitHub:
- Evo: A repository providing code for genome-scale modeling and design, including Colab notebooks for protein folding and CRISPR generation.
- GENA-LM: A family of models optimized for long sequences, with a dedicated web service for user-friendly DNA annotation.
- GPN: A toolkit focused on variant effect prediction in human and plant genomes.
NVIDIA BioNeMo Framework
For industrial-scale applications, the NVIDIA BioNeMo framework offers an optimized environment for training and deploying biomolecular models. BioNeMo includes optimized versions of models like DNABERT and Geneformer, providing curated data loaders and training recipes that are accelerated for NVIDIA GPU clusters. The framework’s NIM (NVIDIA Inference Microservices) allows for rapid deployment, with inference speeds up to 5x faster than baseline implementations.
Applications
DNA language models are becoming accessible to a wide range of users, from academic researchers to large pharmaceutical companies. In academia, most scientists do not train these models from scratch because pretraining requires massive computational resources, often involving hundreds of high-end GPUs running for weeks. Instead, they rely on pretrained foundation models and adapt them to specific biological questions through fine-tuning.
In parallel, pharmaceutical and biotech companies are integrating these systems into drug discovery pipelines. They are using DNA language models to move from traditional search-based drug discovery toward generative design, where models suggest potential therapeutic targets that are then validated in automated laboratory systems. This creates a closed feedback loop between computation and experimentation, significantly reducing development time. In clinical genomics, these models are increasingly used to interpret genetic variants linked to rare diseases. Even with modern sequencing technologies, most patients with suspected rare disorders do not receive a clear molecular diagnosis. DNA language models help prioritize likely disease-causing mutations by evaluating how sequence changes affect predicted biological function. In some cases, AI-assisted systems have already demonstrated higher diagnostic accuracy than traditional workflows, particularly when analyzing complex genomic data.
Beyond clinical and pharmaceutical use, DNA language models are becoming central tools in genomics and precision medicine. One of their key applications is variant effect prediction, where the goal is to determine whether a genetic mutation is harmless or disease-causing. These models learn this directly from sequence patterns without needing explicit experimental labels, which is especially important for the non-coding regions that make up most of the human genome.
Another major application is gene expression modeling, where the system predicts how much protein a gene will produce based on its regulatory DNA. This includes understanding promoters, enhancers, and other regulatory elements that control when genes are active. At the same time, generative models are being used in synthetic biology and CRISPR engineering, where new DNA sequences can be designed with specific functional properties, such as targeted gene activity or organism specificity.
Finally, newer foundation models extend this approach to single-cell and epigenomic data, allowing researchers to study how individual cells differ in gene expression and how cellular states change in diseases like cancer. Together, these applications show that DNA language models are not only tools for analysis, but also for design and intervention across biology.
Future Perspective
The future of DNA language models is moving toward whole-genome reasoning, where systems are no longer limited to local sequence patterns but can interpret and reason across an organism’s entire genetic code. A key direction in this progress is extending context length. Current models typically operate on sequences ranging from 10,000 to 100,000 base pairs, yet many biological regulatory interactions occur across distances measured in millions of base pairs. To address this gap, new architectures are being developed to process ultra-long genomic contexts.
At the same time, the field is moving toward integrating multiple layers of biological information into unified foundation models. Instead of learning only from DNA sequences, future systems will combine genomic data with RNA expression, protein structures, and epigenetic signals, effectively forming a multi-modal “biological model of life.” By training on evolutionary data across thousands of species, these models aim to capture universal biological constraints that govern both human disease and broader biological systems such as crop resilience.
Biology is shifting toward increasingly autonomous discovery workflows. AI systems are beginning to act as agents that not only analyze data but also design, execute, and refine experiments in automated laboratory environments. With the rise of cloud labs and robotic platforms, experimental biology is becoming more programmable, where AI-driven systems can iteratively improve hypotheses and directly guide physical experiments, accelerating discovery from idea to validation.
🤨 Wait… Did you miss the article on Brain-Computer Interfaces?
📰 SUBSCRIBE TO MY LINKEDIN NEWSLETTER
If you liked this newsletter article, I hope you’ll consider being a regular subscriber. 🙏Just scroll up to the very top and click that SUBSCRIBE button.
🪄 My TEDx Talk on BioRevolution is Live!
(English subtitles are available)
Originally published on LinkedIn.