TinyLlama: An Open-Source Small Language Model

By Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu. arXiv.org, 2024

811 citations recorded by the source corpus.

We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of Llama 2, TinyLlama leverages various advances contributed by the open-source community (e.g., FlashAttention and Lit-GPT), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. Our model checkpoints and code are publicly available on GitHub at https://github.com/jzhang38/TinyLlama.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/560c6f24c335c2dd27be0cfa50dbdbb50a9e4bfd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bin Fu
2. Pei Cheng
3. Gang Yu
4. Yuhui Li
5. Fangyun Wei
6. Chao Zhang
7. Rui Wang
8. Hongyang Zhang

Citations in the corpus

• Large Language Models: A Survey (2024, neutral): “1B [121], LLaMA-Pro-8B [122].”
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, background): “An alternative could be to use TinyLLaMA (Zhang et al., 2024), but it is not feasible for instruct-tuned models due to the inconsistency in instruction templates between LLaMA2-Chat and TinyLLaMA-Chat.”
• ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, methodology): “We consider the pre-trained LLM, such as T5 [42], TinyLlama [62] and LLaMA-2 [52], as the text encoder, which provides the comprehensive text feature to condition image generation.”
• Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison (2024, methodology): “For each dataset, we examine their behavior on reward models trained from pre-trained models of different sizes: 350 million (Opt-350m [33]), 1 billion (TinyLlama-1B-3T [34]), and 7 billion parameters (Llama2-7B and Llama2-7B-Chat [2]).”
• BERGEN: A Benchmarking Library for Retrieval-Augmented Generation (2024, unknown): “BERGEN supports the most popular open-weights LLMs such as Llama2 [78], Llama3 [1], SOLAR [38], Mixtral [29], Gemma [76], TinyLlama [91], and Command-R (multilingual).”
• Sparsity-Accelerated Training for Large Language Models (2024, methodology): “1B (Zhang et al., 2024) help us determine (§ 2.2) 1) using maxip (a neuron importance metric) to assess neuron importance, 2) adopting sampling (versus top-k) neuron selection strategy, and 3) introducing a ladder omission rate scheduler.”