OLMo: Accelerating the Science of Language Models

By Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang and 33 more. Annual Meeting of the Association for Computational Linguistics, 2024

700 citations recorded by the source corpus.

Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, we have built OLMo, a competitive, truly Open Language Model, to enable the scientific study of language models. Unlike most prior efforts that have only released model weights and inference code, we release OLMo alongside open training data and training and evaluation code. We hope this release will empower the open research community and inspire a new wave of innovation.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ac45bbf9940512d9d686cf8cd3a95969bc313570

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “Data pruning can have an outsized impact on quality in downstream performance (Marion et al., 2023; Boubdir et al., 2023; Attendu and Corbeil, 2023; Abbas et al., 2024; Groeneveld et al., 2024; Allal et al., 2023; Li et al., 2023c).”
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (2024, background): “Molmo-7B-O and Molmo-7B-D, based on OLMo-7B [10] and Qwen2 7B [33], respectively, perform comfortably between GPT-4V and GPT-4o on both academic benchmarks and user preference.”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, positive): “We use a modified version of the OLMo tokenizer (Groeneveld et al., 2024) which provides better token efficiency and performance on code-related tasks.”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, background): “With the advancement of LMs [4, 35, 42, 45, 13], researchers have demonstrated the possiblity of decomposing complex vision tasks into simpler substeps that can each be solved using vision tools [57, 60, 18, 17].”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, neutral): “…configuration options for each task • Direct access to the specific task formulations used in this work (as well as in earlier work such as OLMo [Groeneveld et al., 2024] and the OLMES standard [Gu et al., 2024]) • Detailed instance-level output data for analysis of model predictions,…”
Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers (2026, unknown): “Across these comparisons, the gains occur at comparable active parameter counts: OLMo-1B and OLMo-MoE both activate 1.28B parameters, LFM2-MoE activates 1.56B compared with 1.17B for LFM2-1.”
Seq vs Seq: An Open Suite of Paired Encoders and Decoders (2025, unknown): “We do so by pulling from the best publicly available datasets used for training decoder-only models, such as Olmo (Groeneveld et al., 2024; OLMo et al., 2025).”
Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison (2024, background): “Reinforcement learning from human feedback (RLHF) is typically the final stage of the modern large language model (LLM) training pipeline [1, 2, 3].”