Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

By Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Hassan Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Singh Behl and 79 more. 2024

2,340 citations recorded by the source corpus.

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, methodology): “We evaluated more than 50 LLMs including open-source and closed-source models, such as GPT-4o [17], Claude-3-Opus [6], and Gemini [30], LLaMA-3 [35], Phi-3 [1] on MMLU-Pro.”
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024, neutral): “17, we demonstrate that modern Language Learning Models (LLMs) such as Gemma2-9B and Phi3-Medium are capable of nearly perfect addition and subtraction up to a length of 4, as well as achieving very high accuracy in multiplication and division.”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, neutral): “The success of Large Language Models (LLMs) [2, 9, 63, 64] and Vision-Language Models (VLMs) [1, 14, 28, 37, 61] has inspired the development of Vision-Language-Action (VLA) models, which extend the capabilities of VLMs by integrating action generation.”
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024, methodology): “Inspired by recent work advocating for training LLMs on educational data [8, 9], we additionally introduce FineWeb-Edu, a subset of 1.3 trillion tokens from FineWeb that was rated as highly educational by a custom classifier.”
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, background): “Released on May 22, 2024, Phi-3-Vision-Instruct [Abdin et al., 2024]”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “As shown, earlier versions primarily built on language models such as InternLM 2 [19], Qwen 2 [268], Phi 3 [1], Yi [279], and Llama 3 [64].”
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, neutral): “5B [Qwen, 2024b], Phi3-mini-Instruct (3B) [Microsoft, 2024, Abdin et al., 2024], Qwen2-Math-7B [Qwen, 2024a] and Qwen2.”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, background): “5-vision-instruct-4B (Abdin et al., 2024), by 3.4% on VideoMME Long subset.”
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, neutral): “Subsequently, we utilize a fast approximation approach to build dynamic sparse masks for different inputs, and then apply these mask to perform sparse attention calculations.”
OmniGen: Unified Image Generation (2024, neutral): “The trans-former model initialized by Phi-3 [1] generates images based on instructions that serve as conditions.”
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, neutral): “5-Vision [AJA + 24] to ensure fair comparisons across all baseline methods.”
Scaling Synthetic Data Creation with 1,000,000,000 Personas (2024, background): “Similar to math problems, logical reasoning problems can also be easily synthesized.”