Qwen2-Audio Technical Report

By Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin and 2 more. arXiv.org, 2024

663 citations recorded by the source corpus.

We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8ca1fdca69d780881f905ef9a4612e62f49a1014

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yifei Xin
2. Sicong Leng
3. Wenqi Zhang
4. Li Bing
5. Shuai Bai
6. Kai Dang
7. Guanzheng Chen
8. Hang Zhang

Citations in the corpus

• Qwen3-Omni Technical Report (2025, neutral): “…reasoning capabilities of unimodal large models (Brown et al., 2020; OpenAI, 2023; Gemini Team, 2024; Anthropic, 2023a;b; 2024; Bai et al., 2023a; Yang et al., 2024; 2025a; Touvron et al., 2023; Dubey et al., 2024; Li et al., 2023; Liu et al., 2023; Zhu et al., 2023; Bai et al., 2023b; 2025; Chu…”
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, neutral): “• Phi-4-Multimodal is the smallest open-sourced multi-modal LLM that behaves better than the open-sourced Qwen2-audio [CXY + 24] with ∼ 2x in size.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “…Zhu et al., 2023b), Linear Projection (Liu et al., 2023b; Chen et al., 2023a) and Dynamic Visual Tokenizer (Jin et al., 2023c), have been widely adopted in popular Video-LLMs (Zhang et al., 2023; Maaz et al., 2023b; Lin et al., 2023a; Li et al., 2023c; Jin et al., 2024; Ataallah et al., 2024).”
• Large Language Diffusion Models (2025, neutral): “Answer: [Yes] Justification: We provide a discussion about societal impacts in the Appendix C. Guidelines: • The answer NA means that there is no societal impact of the work performed.”
• MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, neutral): “Generalized models trained across multiple do-mains—speech, sounds, and music—such as Qwen2-Audio, LTU-AS, and Gemini, exhibit strong overall performance.”
• Qwen2.5-Omni Technical Report (2025, positive): “…general artificial intelligence has become increasingly visible, largely due to advancements in Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; 2024; Gemini Team, 2024; Anthropic, 2023a;b; 2024; Bai et al., 2023a; Yang et al., 2024a; Touvron et al., 2023a;b; Dubey et al., 2024a).”