Textbooks Are All You Need II: phi-1.5 technical report

By Yuan-Fang Li, Sébastien Bubeck, Ronen Eldan, A. Giorno, Suriya Gunasekar, Yin Tat Lee. 2023

664 citations recorded by the source corpus.

We continue the investigation into the power of smaller Transformer-based language models as initiated by \textbf{TinyStories} -- a 10 million parameter model that can produce coherent English -- and the follow-up work on \textbf{phi-1}, a 1.3 billion parameter model with Python coding performance close to the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to generate ``textbook quality"data as a way to enhance the learning process compared to traditional web data. We follow the ``Textbooks Are All You Need"approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named \textbf{phi-1.5}, with performance on natural language tasks comparable to models 5x larger, and surpassing most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding. More generally, \textbf{phi-1.5} exhibits many of the traits of much larger LLMs, both good -- such as the ability to ``think step by step"or perform some rudimentary in-context learning -- and bad, including hallucinations and the potential for toxic and biased generations -- encouragingly though, we are seeing improvement on that front thanks to the absence of web data. We open-source \textbf{phi-1.5} to promote further research on these urgent topics.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Notable examples include the Pile (Gao et al., 2021) and “textbook-like” data sources (Gu-nasekar et al., 2023; Li et al., 2023f).”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “This section details our approach to searching, selecting, and analyzing the relevant literature.”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, neutral): “Recent studies, even though not focused on math, support this view with synthesized textbooks [24, 36].”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, positive): “The developers of Falcon (Penedo et al., 2023) carefully extract high-quality data from the web via heuristic rules and prove that properly curated pertaining corpora lead to powerful LLMs. Li et al. (2023f) propose phi-1.”
Large Language Models: A Survey (2024, neutral): “5 [209], and Phi-2 from Microsoft.”
Proving Test Set Contamination in Black Box Language Models (2023, background): “This setting mirrors many common situations with API-based model providers (Brown et al., 2020b; Bai et al., 2022) and matches an increasing trend where the training data is kept secret for ‘open’ models (Touvron et al., 2023; Li et al., 2023).”
Don't Make Your LLM an Evaluation Benchmark Cheater (2023, methodology): “5 (Li et al., 2023): it is a 1.3B model trained on “textbook quality” data of ≈ 27B tokens, and can achieve comparable performance as much larger models.”
Will we run out of data? Limits of LLM scaling based on human-generated data (2022, background): “For example, differences in data quality (Li et al., 2023) and the number of training epochs (Muennighoff et al., 2023) can potentially have a substantial effect on final model performance.”
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, methodology): “This approach primarily leverages advanced LLMs such as the GPT series (Radford et al., 2019; Brown et al., 2020; OpenAI, 2023) as the guidance to generate high-quality data (Josi-foski et al., 2023; Taori et al., 2023; Chiang et al., 2023; Li et al., 2023).”
LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, background): “On the flip side, many recent works have demonstrated the effectiveness of training on higher quality data during both pre-training (Li et al., 2023d) and fine-tuning (Zhou et al., 2023; Cao et al., 2023) phases.”
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “While most existing works generally target data generation for various tasks (e.g., pre-training (Gunasekar et al., 2023; Li et al., 2023b; Eldan and Li, 2023), fine-tuning (Mukherjee et al., 2023; Mitra et al., 2023; Xu et al., 2023a), evaluation (Feng et al., 2023; Wei et al., 2024)) across…”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, methodology): “In our previous works on the phi models [GZA + 23, LBE + 23, JBA + 23] it was shown that a combination of LLM-based filtering of publicly available web data, and LLM-created synthetic data, enable performance in smaller language models that were typically seen only in much larger models.”
Phi-4 Technical Report (2024, positive): “Building on the success of the Phi family [GZA + 23, LBE + 23, JBA + 23, AAA + 24], we introduce phi-4 , a 14-billion parameter model that further advances performance of small language models by introducing innovative synthetic data generation methods for reasoning-focused tasks, by optimizing the…”