Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

By Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen. International Conference on Learning Representations, 2023

500 citations recorded by the source corpus.

The popularity of LLaMA (Touvron et al., 2023a;b) and other recently emerged moderate-sized large language models (LLMs) highlights the potential of building smaller yet powerful LLMs. Regardless, the cost of training such models from scratch on trillions of tokens remains high. In this work, we study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains. We demonstrate the efficacy of our approach by presenting the Sheared-LLaMA series, pruning the LLaMA2-7B model down to 1.3B and 2.7B parameters. Sheared-LLaMA models outperform state-of-the-art open-source models of equivalent sizes, such as Pythia, INCITE, OpenLLaMA and the concurrent TinyLlama models, on a wide range of downstream and instruction tuning evaluations, while requiring only 3% of compute compared to training such models from scratch. This work provides compelling evidence that leveraging existing LLMs with structured pruning is a far more cost-effective approach for building competitive small-scale LLMs

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/abdb0f9d1486dbb024c4bc9f8f9dc40464c58715

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yung-Sung Chuang
2. Pengcheng He
3. Yoon Kim
4. James R. Glass
5. Yaojie Lu
6. Hongyin Luo
7. Hongyu Lin
8. Mingyu Xu

Citations in the corpus

• A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “For example, Li et al. [63] score the filters by calculating the sum of the absolute magnitude of their weights.”
• LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, unknown): “3B , Xia et al., 2023), Llama-2-7B-chat ( LLaMA-2-7B , Touvron et al., 2023), and Mistral-7B-Instruct-v0.”
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models (2023, methodology): “We experiment with OpenLLaMa (Geng & Liu, 2023) and Sheared-LLaMA (Xia et al., 2023) models in the size of 7B, 3B, 2.7B, 1.3B.”
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “ShearedLLaMA Xia et al. (2023) employs targeted structured pruning and dynamic batch loading.”
• How Can Recommender Systems Benefit from Large Language Models: A Survey (2023, unknown): “Besides, FaiR-LLM [Zhang et al. , 2023a] and UP5 [Hua et al. , 2023a] intend to address the fairness issue when adapting LLM for item generation tasks.”
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025, unknown): “In our experiments, we set k = 20 , n = 10 .”