Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

By Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen and 4 more. arXiv.org, 2024

713 citations recorded by the source corpus.

Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8dc5a5f57b5a4564536badf3ca98e5680f313314

Citations in the corpus

• ProtoSiTex: Learning Semi-Interpretable Prototypes for Multi-label Text Classification (2025, unknown): “…from Table III, compared with traditional, graph-based, and advanced black-box models, ProtoSiTex demonstrates consistent performance improvements, highlighting the limitations of static embeddings, recurrent encoders, and global representation learning in capturing contextual semantics and…”
• Mixture of Languages: Improved Multilingual Encoders Through Language Grouping (2025, unknown): “Recently, monolingual encoders have received increasing interest, marked by the releases of Mod-ernBERT (Warner et al., 2024) and NeoBERT (Bre-ton et al., 2025).”
• Embedding Models Measure in Peculiar Ways (2026, unknown): “ModernBERT-based encoders.”
• Latent Trajectory Discrimination for AI-Generated Text Detection (2026, unknown): “More recent models, such as ModernBERT [5, 37], improve long context handling and computational efficiency.”
• Closing the Operational Gap in Semantic Caching (2026, unknown): “We fine-tune Lang-Cache Reranker models, domain-specific cross-encoders initialized from GTE-Reranker-ModernBERT-base (Zhang et al., 2024; Li et al., 2023; Warner et al., 2025”
• Granite Embedding Multilingual R2 Models (2026, unknown): “In late April 2026, we observed that our ModernBERT-based embedding models had roughly halved in throughput, with no corresponding change in our own training or inference code.”
• ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models (2026, unknown): “Despite being very helpful, this mechanism is not used in modern models leveraging Flash Attention 3 [15] such as ModernBERT [16], the base model used by all models in our experiments.”
• Avey-B (2026, unknown): “…a bidirectional reformulation of Avey for the encoder-only setting, and compare it empirically against widely used and recently introduced Transformer-based encoders, namely, BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), ModernBERT (Warner et al., 2025), and NeoBERT (Breton et al., 2025).”
• HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation (2026, unknown): “For HyTE-H bert , which is fine-tuned with modernbert-base (Warner et al., 2024) as base model, we compare against state-of-the-art embedding models smaller than 500M parameters, including gte-multilingual-base (Zhang et al., 2024), KaLM-embedding-multilingual-mini-v1 (Hu et al., 2025), and…”
• EmbeddingRWKV: State-Centric Retrieval with Reusable States (2026, unknown): “We additionally compare against three Transformer-based rerankers, including Modern-BERT (Warner et al., 2024), EuroBERT (Boizard et al., 2025), and the Jina reranker (Wang et al., 2025).”
• A Large-Language-Model Framework for Automated Humanitarian Situation Reporting (2025, unknown): “We embed the normalized paragraphs into dense vector representations employing the 768-dimensional embeddings produced by ModernBERT 24 , a more efficient and higher-performing transformer encoder than the standard BERT model.”
• Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report (2025, unknown): “On the other hand, virtually all Large Language Models employ some form of casing, which ModernBERT [33], and thus subsequently Ettin [35], also adopts.”
• ModernVBERT: Towards Smaller Visual Document Retrievers (2025, unknown): “It combines a state-of-the-art 150M text bidirectional encoder (Weller et al., 2025) with the ModernBERT architecture (Warner et al., 2024a) and a small vision encoder SigLIP2-16B-512 of 100M parameters (Tschannen et al., 2025).”
• Turk-LettuceDetect: A Hallucination Detection Model for Turkish RAG Applications (2025, unknown): “ModernBERT [12], building upon BERT’s foundational design [20], incorporates several innovations such as rotary positional embeddings (RoPE) [26] and local-global attention mechanisms, enabling it to process sequences up to 8,192 tokens.”
• mmBERT: A Modern Multilingual Encoder with Annealed Language Learning (2025, unknown): “5 For English, we also show results with ModernBERT (Warner et al., 2024) as an upper bound.”
• Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs (2025, unknown): “There has also been a recent uptick in BERT-style embedding models including Nomic Embed (Nussbaum et al., 2024), Arctic-Embed (Merrick et al., 2024), and ModernBERT (Warner et al., 2024).”