DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome

By Yanrong Ji, Zhihan Zhou, Han Liu, Ramana V. Davuluri. bioRxiv, 2020

1,273 citations recorded by the source corpus.

Deciphering the language of non-coding DNA is one of the fundamental problems in genome research. Gene regulatory code is highly complex due to the existence of polysemy and distant semantic relationship, which previous informatics methods often fail to capture especially in data-scarce scenarios. To address this challenge, we developed a novel pre-trained bidirectional encoder representation, named DNABERT, that forms global and transferrable understanding of genomic DNA sequences based on up and downstream nucleotide contexts. We show that the single pre-trained transformers model can simultaneously achieve state-of-the-art performance on many sequence predictions tasks, after easy fine-tuning using small task-specific data. Further, DNABERT enables direct visualization of nucleotide-level importance and semantic relationship within input sequences for better interpretability and accurate identification of conserved sequence motifs and functional genetic variants. Finally, we demonstrate that pre-trained DNABERT with human genome can even be readily applied to other organisms with exceptional performance.

Subfields: Molecular Biology

View Paper: https://www.semanticscholar.org/paper/c43d9cade31600400a0f62beb5bbcc1b548e009e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

SBSM-Pro: support bio-sequence machine for proteins (2023, neutral): “Platforms such as Kipoi [22], Pysster [23], Selene [24], and DNA-BERT [25] have been developed for deep learning-based classiﬁcation of biological sequences.”
The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics (2024, background): “5 billion parameters, which is five times larger than DNABert [18] and ten times larger than the Enformer [14].”
Exploring the Limits of Out-of-Distribution Detection (2021, background): “…[Elnaggar et al., 2020, Dohan et al., 2021, Littmann et al., 2021], protein structure prediction [Rives et al., 2021], and predicting promoters, splice sites and transcription factor binding sites [Ji et al., 2020], but it has not yet been studied how pre-training could help near-OOD detection.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “These tasks include predicting genome-wide variant effects (GPN [116]; DNABERT [117]; and its subsequent evolution, DNABERT-2 [118]), predicting DNA cis-regulatory regions (DNAGPT [119], DNABERT, and DNABERT-2), predicting DNA-protein interactions (TFBert [120] and MoDNA [121]), and determining RNA…”
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, methodology): “These tokenizers use fixed k-mers (short overlapping sequences of length k) or frequency-based byte pair encoding (BPE), that attempt to aggregate DNA into meaningful units [25, 55].”
The language of proteins: NLP, machine learning & protein sequences (2021, background): “Deep language models have started to show promise in protein and genomic research [119,45,83,79,29,107,34,40].”
DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome (2023, methodology): “The construction of the benchmark starts with the aggregation of various biologically important genome analysis datasets, followed by the assessment of existing models such as DNABERT (Ji et al., 2021) and Nucleotide Transformer (Dalla-Torre et al., 2023) on these datasets.”