Qwen2 Technical Report

By An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang and 48 more. 2024

2,333 citations recorded by the source corpus.

This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models. We release a comprehensive suite of foundational and instruction-tuned language models, encompassing a parameter range from 0.5 to 72 billion, featuring dense models and a Mixture-of-Experts model. Qwen2 surpasses most prior open-weight models, including its predecessor Qwen1.5, and exhibits competitive performance relative to proprietary models across diverse benchmarks on language understanding, generation, multilingual proficiency, coding, mathematics, and reasoning. The flagship model, Qwen2-72B, showcases remarkable performance: 84.2 on MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH as a base language model. The instruction-tuned variant, Qwen2-72B-Instruct, attains 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench. Moreover, Qwen2 demonstrates robust multilingual capabilities, proficient in approximately 30 languages, spanning English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai, Vietnamese, and more, underscoring its versatility and global reach. To foster community innovation and accessibility, we have made the Qwen2 model weights openly available on Hugging Face and ModelScope, and the supplementary materials including example code on GitHub. These platforms also include resources for quantization, fine-tuning, and deployment, facilitating a wide range of applications and research endeavors.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ziwei Liu
2. Wenhu Chen
3. Bo Li
4. Yu Su
5. Graham Neubig
6. Huan Sun
7. Fei Huang
8. Chujie Zheng

In Computer Vision and Pattern Recognition

1. Bo Li
2. Graham Neubig
3. Ziwei Liu
4. Jingren Zhou
5. Luc Van Gool
6. Xiaogang Wang
7. Wenhu Chen
8. Yuanhan Zhang

In Large Language Models

1. Wenhu Chen
2. Ziwei Liu
3. Huan Sun
4. Bo Li
5. Fei Huang
6. Chujie Zheng
7. Haiyang Xu
8. Graham Neubig

Citations in the corpus

• Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, neutral): “…popularity of Large Language Models (LLMs) such as GPT (Radford et al., 2018, 2019; Brown et al., 2020), Llama (Touvron et al., 2023; Dubey et al., 2024), and Qwen (Bai et al., 2023; Yang et al., 2024), encoder-only models remain widely used in a variety of non-generative downstream applications.”
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, neutral): “Specifically, we select 300K instances from M AGPIE -Pro and M AGPIE -Air-Filtered, yielding datasets M AGPIE -Pro-300K and M AGPIE -Air-300K-Filtered, respectively.”
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “…LAMM (Yin et al., 2023b), LVLM-eHub (Xu et al., 2023), SEED (Li et al., 2024a), MMBench (Liu et al., 2023d),CV-Bench (Tong et al., 2024a), MM-Vet (Yu et al., 2024), Mantis (Jiang et al., 2024), and BLINK (Fu et al., 2024) have emerged, covering aspects from basic perception to hallucination…”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “As shown, earlier versions primarily built on language models such as InternLM 2 [19], Qwen 2 [268], Phi 3 [1], Yi [279], and Llama 3 [64].”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “We choose Qwen-2 [147] as our LLM f ϕ ( · ) parameterized by ϕ , as it offers various model size and exhibits strong language capabilities to date among publicly available checkpoints.”
• Video Instruction Tuning With Synthetic Data (2024, positive): “This issue is exacerbated when using large-parameter LLMs; with the Qwen2-72B model, we could only process 8 frames before maxing out the memory on 128 NVIDIA H100 GPUs.”
• mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “Specifically, for Qwen2, we select layers [0 , 9 , 17 , 25] .”
• Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, neutral): “(Vavekanand & Sam, 2024), Gemma2 (9B, 27B) (Team et al., 2024), Mixtral (8x7B) (Jiang et al., 2024), and Qwen2 (7B, 72B) (Yang et al., 2024a), and the closed-source ones are GPT (3.5-Turbo, 4o, 4o-mini) (OpenAI, 2022; 2024b) and Claude-3.”
• WebThinker: Empowering Large Reasoning Models with Deep Research Capability (2025, neutral): “Large reasoning models (LRMs) enhance test-time performance through extended reasoning, setting them apart from traditional large pre-trained models that scale mainly with model size or data volume [16, 62, 40, 29, 3].”
• Graph Retrieval-Augmented Generation: A Survey (2024, neutral): “Then, we provide formal definitions for two types of models that can be used in the retrieval and generation stages: Graph Neural Networks and Language Models.”
• Qwen3 Technical Report (2025, positive): “Besides, we remove QKV-bias used in Qwen2 (Yang et al., 2024a) and introduce QK-Norm (Dehghani et al., 2023) to the attention mechanism to ensure stable training for Qwen3.”
• The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025, neutral): “In recent years, Large Language Models (LLMs) have made remarkable advances in mathematical reasoning (OpenAI, 2023; Dubey et al., 2024; Shao et al., 2024; Zhu et al., 2024; Yang et al., 2024a;c;b), yet they can make mistakes, such as miscalculations or logical errors, leading to wrong conclusions.”
• Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA (2026, unknown): “A recent evaluation benchmark, HKCanto-Eval (Cheng et al., 2025), also shows that Qwen2-72B (Yang et al., 2024) outperforms most models on academic, professional, and cultural QA for Cantonese.”
• ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows (2025, unknown): “For supervised fine-tuning (SFT), we employ the LLaMA-Factory framework (Zheng et al., 2024), training for three epochs and selecting the checkpoint with the lowest validation loss.”
• R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training (2025, unknown): “5B model [29] as an illustrative example.”