Textbooks Are All You Need

By Suriya Gunasekar, Yi Zhang, Jyoti Aneja, C. C. T. Mendes, A. Giorno, S. Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi and 9 more. 2023

626 citations recorded by the source corpus.

We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality"data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, methodology): “…2023b) and CodeGen 2 (Nijkamp et al., 2023a), GPT-NeoX (Black et al., 2022), SantaCoder (Allal et al., 2023), StarCoder (Li et al., 2023) and phi-1 (Gunasekar et al., 2023), consistently demonstrating better performance on code benchmarks than general-purpose LLMs of comparable or even larger size.”
Large Language Models for Software Engineering: Survey and Open Problems (2023, result): “[53] reported that, by training with only a textbook-quality code corpus, LLMs with lower parameter counts could achieve performance comparable to much larger models.”
A Survey on Large Language Models for Code Generation (2024, neutral): “between smaller and larger LLMs. 7B/13B models outperform the 70B model.”
Large Language Models: A Survey (2024, positive): “Prominent works in this direction include Phi-1 [208], Phi-1.”
VerilogEval: Evaluating Large Language Models for Verilog Code Generation (2023, methodology): “Additionally, a recent study utilized GPT-4 to generate a high-quality synthetic textbook dataset, achieving superior coding capabilities at 1/100th of the cost of other models [20].”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “The Phi series models (Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023) prioritize data quality and employ synthetic methods to generate data of “textbook quality” to enhance the learning experience for smaller models.”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “The overall training of MiniCPM base model includes three stages: stable training stage, decay stage, SFT stage (Zhang et al., 2023; Wei et al., 2021).”
Improving Text Embeddings with Large Language Models (2023, methodology): “Unnatural Instructions [15] is a synthetic instruction following dataset by prompting existing LLMs. Orca [28] and Phi [14] propose to train better small language models by using high-quality synthetic data from GPT-3.”
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024, methodology): “Inspired by recent work advocating for training LLMs on educational data [8, 9], we additionally introduce FineWeb-Edu, a subset of 1.3 trillion tokens from FineWeb that was rated as highly educational by a custom classifier.”
Large Language Models for Information Retrieval: A Survey (2023, neutral): “This intriguing area remains open for further investigation and advancement.”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, neutral): “Modern methods typically leverage off-the-shelf Large Language Models (LLMs) [5], [6] for scene descriptions to generate object affordance, location, or heatmaps, followed by a predeﬁned motion planner to complete the tasks [7], [8].”
Textbooks Are All You Need II: phi-1.5 technical report (2023, methodology): “To do so we create a dataset of 95B tokens of filtered web data following the filtering technique in [GZA + 23].”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, positive): “3 It is an open-source counterpart to the Phi work [24].”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, background): “For example, a study by Gunasekar et al. [43] demonstrated that a Transformer-based model with only 1.3 billion parameters could achieve commendable performance on code generation tasks when trained on 6 billion tokens of “textbook quality” web data and an additional 1 billion tokens generated…”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “…task-solving capability, including curation of larger and higher-quality general or domain-specific data (Touvron et al., 2023a; Zhou et al., 2023; Gunasekar et al., 2023; Feng et al., 2023; Taylor et al., 2022), more sophisticated prompt design (Wei et al., 2022; Zhou et al., 2022; Creswell et…”
Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation (2024, background): “The expertise that behavioral health experts could draw on to generate instructive examples and curate high-quality datasets holds particular value in light of recent evidence that quality of data trumps quantity of data for training well-performing models 66 .”
Phi-4 Technical Report (2024, positive): “Building on the success of the Phi family [GZA + 23, LBE + 23, JBA + 23, AAA + 24], we introduce phi-4 , a 14-billion parameter model that further advances performance of small language models by introducing innovative synthetic data generation methods for reasoning-focused tasks, by optimizing the…”