mmBERT: A Modern Multilingual Encoder with Annealed Language Learning

By Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn J. Lawrie, Benjamin Van Durme. 2025

Subfields: Natural Language Processing, Representation & Self-Supervised Learning, Model Architectures

Citations in the corpus

• Granite Embedding Multilingual R2 Models (2026, unknown): “Following the mmBERT approach (Marone et al., 2025), we train three variants during this stage and linear merge them to form the final model: (a) an English-focused variant with higher proportions of English data (b) a multilingual-focused variant trained on data spanning more than 1,800 languages…”
• Seq vs Seq: An Open Suite of Paired Encoders and Decoders (2025, unknown): “Our work shows that at scale, for SOTA models, encoders do greatly outperform encoders-from-decoders (as also shown by SOTA concurrent work Marone et al. (2025)).”
• ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance (2025, unknown): “Continuous architectural and training objective improvements have led to more performant and efficient encoder-only transformer variants, among which DeBERTaV3 (He et al., 2021a) and the recently proposed ModernBERT (Warner et al., 2024) stand out as major improvements.”
• DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search (2026, unknown): “We train paired dense and late-interaction models with multilingual data using the same 307M-parameter mmBERT-base (Marone et al., 2025) backbone.”