Qwen Technical Report

By Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhang Ge, Fei Huang, Binyuan Hui and 39 more. 2023

4,095 citations recorded by the source corpus.

Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans. In this work, we introduce Qwen, the first installment of our large language model series. Qwen is a comprehensive language model series that encompasses distinct models with varying parameter counts. It includes Qwen, the base pretrained language models, and Qwen-Chat, the chat models finetuned with human alignment techniques. The base language models consistently demonstrate superior performance across a multitude of downstream tasks, and the chat models, particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are highly competitive. The chat models possess advanced tool-use and planning capabilities for creating agent applications, showcasing impressive performance even when compared to bigger models on complex tasks like utilizing a code interpreter. Furthermore, we have developed coding-specialized models, Code-Qwen and Code-Qwen-Chat, as well as mathematics-focused models, Math-Qwen-Chat, which are built upon base language models. These models demonstrate significantly improved performance in comparison with open-source models, and slightly fall behind the proprietary models.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Ping Luo
3. Yu Qiao
4. Mohit Bansal
5. Sergey Levine
6. Percy Liang
7. Maosong Sun
8. Zhiyuan Liu

In Large Language Models

1. Hong-Liu Cao
2. Wenhu Chen
3. Rafael Rafailov
4. Siddharth Karamcheti
5. Damai Dai
6. Bo Li
7. Fei Huang
8. Yu Qiao

Citations in the corpus

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “• Open-source models include: general models like (1) DeepSeek-LLM-Chat 67B (DeepSeek-AI, 2024), (2) Qwen 72B (Bai et al., 2023), and (3) ChatGLM3 6B (ChatGLM3 Team, 2023), as well as models with enhancements in mathematics including (4) InternLM2-Math 20B 6 which builds on InternLM2 and underwent…”
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “The advent of GPT series [48, 49, 9, 45, 1] and other autoregressive (AR) large language models (LLMs) [13, 3, 27, 63, 64, 70, 60, 4, 61] has heralded a new epoch in the field of artificial intelligence.”
• How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “Large language models (LLMs) [3, 6, 9, 10, 13, 26, 99, 101, 103, 107, 108, 117, 118, 136] have greatly advanced AGI by enabling complex language tasks previously thought human-exclusive.”
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, neutral): “…GPT-4-based judge can judge LLMs like a human expert, much concurrent work delving into curating training data (Li et al., 2023a; Kim et al., 2023; Cui et al., 2023; Xu et al., 2023; Jiang et al., 2023; Wang et al., 2023) and benchmarks (Lambert et al., 2024; Zheng et al., 2023) with the help of…”
• Qwen2.5-Coder Technical Report (2024, background): “5 (Bai et al., 2023a), and Code-Stral (team, 2024), have demonstrated superior performance in coding evaluations (Chen et al., 2021; Austin et al., 2021; Cassano et al., 2022; Jain et al., 2024).”
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2024, methodology): “We can find that our model has a higher safety score than both ChatGPT and GPT-4, placing it amongst the ranks of the safest models.”
• Qwen3 Technical Report (2025, positive): “Qwen3 models utilize Qwen’s tokenizer (Bai et al., 2023), which implements byte-level byte-pair encoding (BBPE, Brown et al., 2020; Wang et al., 2020; Sennrich et al., 2016) with a vocabulary size of 151,669.”
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “UltraFeedback (Cui et al., 2023), consisting of 5 K prompts, as another out-of-distribution (OOD) test set.”
• LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, neutral): “For example, the commonly used CLIP visual encoder only has 0.3B parameters, while the corresponding LLM such as LLaMA [43], Vicuna [45], DeepSeek [27], and Qwen [3] can have 7B, 13B, or even 70B+ parameters.”
• RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “Many popular models trained with RLHF use new preference datasets such as UltraFeedback (Cui et al., 2023) or Nectar (Zhu et al., 2023a Additional data is included in the code-base, but not included in the evaluation score due to noisy results or lack of clear use instructions (e.g. could be easy fo”
• OpenVLA: An Open-Source Vision-Language-Action Model (2024, background): “…the key advances fueling recent VLMs are model architectures that bridge features from pretrained vision encoders [8, 9, 25] with pretrained language models [10, 23, 34– 36], directly building on advances in both computer vision and natural language modelling to create powerful multimodal models.”
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, methodology): “For architecture designs, we follow LLaVA-NeXT [24] to adopt the most general framework, i.e., a vision encoder [62], an intermediate projector, and a powerful LLM [4].”
• Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “The progress in the field has been further accelerated by the emergence of open-source LLMs, including the LLaMA series [106, 107], Vicuna [145], InternLM [104], MOSS [101], ChatGLM [36], Qwen [4], Baichuan [6], and Falcon [86], among others [26, 103, 119].”
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, background): “…human-like conversational manner [Ouyang et al. 2022; OpenAI 2022, 2023b; Google 2023; Anthropic 2023; Workshop et al. 2022; Touvron et al. 2023a,b; Bai et al. 2023a; Yang et al. 2023; Team 2023; Bi et al. 2024] and incredible scalability [Kaplan et al. 2020; Henighan et al. 2020; Hoffmann et al.…”
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “5-7B [63], Yi-1.”