Yi: Open Foundation Models by 01.AI

By 01.AI Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jingwen Chang and 20 more. 2024

869 citations recorded by the source corpus.

We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Model Architectures

Citations in the corpus

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, neutral): “We employ eight models: Llama-2-7B, Llama-2-7B-Chat, Llama-2-13B, Llama-2-13B-Chat [35], Mistral-7B [20], Gemma-7B [34], Yi-6B, and Yi-6B-Chat [43].”
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, neutral): “Additionally, we observe that Llama3-8B achieves the best performance among these models, while Yi-6B (Young et al., 2024) and Mistral-7B (Jiang et al., 2023a) exhibit competitive performance among models of the same size.”
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024, background): “Previous works (Young et al., 2024; Zhou et al., 2024) argue that fewer than 10K instances of SFT data are enough to produce satisfactory results.”
Scaling Synthetic Data Creation with 1,000,000,000 Personas (2024, methodology): “We present examples of typical logical reasoning problems synthesized using our proposed persona-driven methodology in Figure 11.”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “5-34B-Chat (Young et al., 2024), and Mixtral-8 × 22B-Instruct (gpt-4o)—and tasked them with answering the MMMU questions without access to images.”
RULER: What's the Real Context Size of Your Long-Context Language Models? (2024, background): “Recent advancements in AI system engineering (Dao et al., 2022; Jacobs et al., 2023; Fu et al., 2024) and language model designs (Chen et al., 2023; Xiong et al., 2023) have enabled efficient scaling up of context length for language models (Liu et al., 2024a; Young et al., 2024).”
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, neutral): “…infrastructures and length generalization methods, several LLMs have pur-portedly managed to process data exceeding 100K tokens (Peng et al., 2023b; OpenAI, 2023b; 01.AI, 2023b,a), with Claude 2 (Anthropic, 2023) and Kimi-Chat (AI, 2023) even claiming to be able to process up to 200K tokens.”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “Recent advancements in multi-modal learning, driven by large language models, have led to progress in image-text dialogue[35, 10, 16, 56, 63, 13, 2, 65] and text-to-image generation tasks[3, 52, 6, 53, 70, 8].”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “5-34B [64] and MoE models: DeepSeek-MoE-16B [65] and Mixtral-8x7B [66].”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, neutral): “Blind LLMs w/ Language Referral.”
LiveBench: A Challenging, Contamination-Free LLM Benchmark (2024, neutral): “[64], zephyr-7b-alpha , and zephyr-7b-beta [60].”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “Furthermore, as indicated in Yi Tech Report (Young et al., 2024) and Zebra (Song et al., 2023), we use of synthetic long QA data that significantly enhances model performance in context-aware tasks.”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, neutral): “5-13B [142], and Hermes-2-Yi-34B [130].”
Qwen2 Technical Report (2024, background): “5-34B (Young et al., 2024) Table 3: Performance of the 30B+ dense models and 40B+ MoE models.”