LLaMA: Open and Efficient Foundation Language Models
By Hugo Touvron, Thibaut Lavril, Gautier Izacard, X. Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Naman Goyal, Eric Hambro and 5 more. arXiv.org, 2023
20,887 citations recorded by the source corpus.
We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
Subfields: Artificial Intelligence, Large Language Models
View Paper: https://arxiv.org/abs/2302.13971
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Graham Neubig
- Yoshua Bengio
- Quoc V. Le
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Furu Wei
- Noah A. Smith
- Jianfeng Gao
- Graham Neubig
- Hinrich Schütze
- Hannaneh Hajishirzi
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Notable in this progression is the rise of Llama, recognized for its focus on computational efficiency during inference (Touvron et al., 2023).”
- Mistral 7B (2023, methodology): “Mistral 7B outperforms the previous best 13B model (Llama 2, [26]) across all tested benchmarks, and surpasses the best 34B model (LLaMa 34B, [25]) in mathematics and code generation.”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “Large unsupervised language models (LMs) trained on very large datasets acquire surprising capabilities [11, 7, 40, 8].”
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “As LLMs are widely adopted, models trained past the recommendations of scaling laws are bound to become increasingly common to amortize inference costs (Touvron et al., 2023).”
- The False Promise of Imitating Proprietary LLMs (2023, methodology): “We consider decoder-only models ranging in size from 1.5B to 13B parameters: GPT-2 1.5B (Radford et al., 2019), LLaMA 7B (Touvron et al., 2023), and LLaMA 13B. 2 We also study the effect by data scale by fine-tuning with different sized data subsets.”
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, background): “Moreover, while some language models already use multi-query attention, such as PaLM (Chowdhery et al., 2022), many do not, including publicly available language models such as T5 (Raffel et al., 2020) and LLaMA (Touvron et al., 2023).”
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, methodology): “Although models have for some time exhibited an extraordinary ability to fit the training data and generalize based on their trained objective [2, 3], their adoption among the general public has until recently been slow.”
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “, 2022), LLaMa-65B (Touvron et al., 2023) and InternLM-104B (SenseTime, 2023)) on AGIEval.”
- Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “We also experiment with Vicuna-13b (Chiang et al., 2023), a version of LLaMA-13b (Touvron et al., 2023) fine-tuned on conversations sourced from the web.”
- Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, methodology): “In this work, we build vLLM , a high-throughput distributed LLM serving engine on top of PagedAttention that achieves near-zero waste in KV cache memory. vLLM uses block-level memory management and preemptive request scheduling that are co-designed with PagedAttention. vLLM supports popular LLMs suc”
- Code Llama: Open Foundation Models for Code (2023, methodology): “ChatGPT and GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022) and Chinchilla (Hoffmann et al., 2022) are closed source, while BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022b), and the seminal work of Llama are public (Touvron et al., 2023a).”
- Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “However, the rapid scientific progress in this area would be impossible without open-access LLMs such as LLaMA 1 and 2 (Touvron et al., 2023), Falcon (TII UAE, 2023), BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022), or NeoX/Pythia (Biderman et al., 2023).”
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “5, Claude-V1, Vicuna-13B, Alpaca-13B [38], and LLaMA-13B [39].”
- Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, background): “Furthermore, it is possible to obtain the same reduction in memory usage since all tokens in a block (except the landmark itself) can be swapped out and only loaded when the corresponding landmark token is activated (see Appendix G).”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “To additionally study the advantages of NF4 over other 4-bit data types, we use the setup of Dettmers and Zettlemoyer [13] and measure post-quantization zero-shot accuracy and perplexity across different models (OPT [73], LLaMA [58], BLOOM [53], Pythia [7]) for model sizes 125m - 13B.”
- Visual Instruction Tuning (2023, background): “Among them, LLaMA [49] is an open-source LLM that matches the performance of GPT-3.”
- StarCoder: may the source be with you! (2023, methodology): “…other open-access models exist, the level of openness still varies across these projects; and some models with released weights have restrictions on model distribution (Touvron et al., 2023), or do not release their training datasets (Nijkamp et al., 2023; Zhang et al., 2022; Fried et al., 2022).”
- Stable and low-precision training for large-scale vision-language models (2023, methodology): “As such, we may have different failure modes for trans-formers pretrained on text, such as GPT-3 [5] or LLaMA [59], compared to CLIP models [46].”
- Instruction Tuning with GPT-4 (2023, background): “LLaMA (Touvron et al., 2023) is a series of open-sourced LLMs, which match the performance of proprietary LLMs such as GPT-3.”