The Falcon Series of Open Language Models

By Ebtesam Almazrouei, Hamza Alobeidli, A. Alshamsi, Alessandro Cappelli, Ruxandra-Aimée Cojocaru, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune and 2 more. arXiv.org, 2023

738 citations recorded by the source corpus.

We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest model, Falcon-180B, has been trained on over 3.5 trillion tokens of text--the largest openly documented pretraining run. Falcon-180B significantly outperforms models such as PaLM or Chinchilla, and improves upon concurrently developed models such as LLaMA 2 or Inflection-1. It nears the performance of PaLM-2-Large at a reduced pretraining and inference cost, making it, to our knowledge, one of the three best language models in the world along with GPT-4 and PaLM-2-Large. We report detailed evaluations, as well as a deep dive into the methods and custom tooling employed to pretrain Falcon. Notably, we report on our custom distributed training codebase, allowing us to efficiently pretrain these models on up to 4,096 A100s on cloud AWS infrastructure with limited interconnect. We release a 600B tokens extract of our web dataset, as well as the Falcon-7/40/180B models under a permissive license to foster open-science and accelerate the development of an open ecosystem of large language models.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/2c0312c604f9f7638bb4533b39e0ae81e7f6ab12

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

OLMo: Accelerating the Science of Language Models (2024, methodology): “This leaves C4 (Raffel et al., 2020), mC4-en (Chung et al., 2023), Wikitext 103 (Merity et al., 2016), Penn Treebank (Marcus et al., 1999; Nunes, 2020), RedPajama (Together Computer, 2023), Falcon-RefinedWeb (Penedo et al., 2023), Dolma (Soldaini et al., 2024), M2D2 S2ORC (Reid et al., 2022), M2D2 W”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “…used learning rate strategy is the Cosine LRS (Kaplan et al., 2020; Hoffmann et al., 2022; Rae et al., 2021; Touvron et al., 2023; Bai et al., 2023; Almazrouei et al., 2023), which gradually decreases the learning rate following a cosine curve after it reaches its maximum after the warmup stage.”
Understanding generative AI output with embedding models (2025, unknown): “We tasked three LLMs— Llama-2 70B ( 45 ), Mixtral 8x7B ( 46 ), and Falcon 40B ( 47 )—with generating responses to questions posted on Stack Exchange ( 48 ).”
“Small” Large Language Models in the Hospital: Evaluation Study on Real-World Data in a Resource-Constrained Setting (2025, unknown): “Based on these criteria, we selected the following 6 open-source models for evaluation: Mistral-Small-24B-Instruct-2501 [18], Phi4-14B [19], Falcon3-10B-Instruct [20], Llama3.”
Understanding Generative AI Content with Embedding Models (2024, methodology): “It is, however, known that data scrapings from StackExchange and the arXiv are used to train and assess LLMs [39, 40].”
Mobile Game Evaluation Method Based on Data Mining of Affective Time Series (2025, unknown): “As such, it allows for exceptions to the separation criterion: The prime example is the tutorial, which typically should overlap with most of the other GAIMs in order to provide the player with a comprehensive introduction to all the elements of the gameplay.”