Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models

By Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, Jingren Zhou. arXiv.org, 2023

809 citations recorded by the source corpus.

Recently, instruction-following audio-language models have received broad attention for audio interaction with humans. However, the absence of pre-trained audio models capable of handling diverse audio types and tasks has hindered progress in this field. Consequently, most existing works have only been able to support a limited range of interaction capabilities. In this paper, we develop the Qwen-Audio model and address this limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types, such as human speech, natural sounds, music, and songs, to facilitate universal audio understanding abilities. However, directly co-training all tasks and datasets can lead to interference issues, as the textual labels associated with different datasets exhibit considerable variations due to differences in task focus, language, granularity of annotation, and text structure. To overcome the one-to-many interference, we carefully design a multi-task training framework by conditioning on a sequence of hierarchical tags to the decoder for encouraging knowledge sharing and avoiding interference through shared and specified tags respectively. Remarkably, Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Building upon the capabilities of Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/f90595f99a0c66d2bb6d0f230f17c7cd8c58f44d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yuhang Zang
2. Fei Huang
3. Binyuan Hui
4. Jingren Zhou
5. Sinan Tan
6. Sicong Leng
7. Xiao-wen Dong
8. Xiaohuan Zhou

In Signal Processing

1. Jingren Zhou
2. Junyang Lin
3. Rui Men
4. Jianxin Ma

Citations in the corpus

• Qwen2 Technical Report (2024, methodology): “Additionally, we assess the performance of human preference alignment and instruction following by evaluating on benchmarks including MT-Bench (Zheng et al., 2023), Arena-Hard (Li et al., 2024), AlignBench (Liu et al., 2023b), MixEval (Ni et al., 2024) whose results approximate those of Chatbot Aren”
• VLMEvalKit: An Open-Source ToolKit for Evaluating Large Multi-Modality Models (2024, background): “Most of the research has focused on LMMs for image and text [5,29], but research has also been extended to other modalities, such as audiotext [10], video [7, 25, 40], or point clouds [46, 47].”
• Qwen3-Omni Technical Report (2025, positive): “…al., 2023a; Yang et al., 2024; 2025a; Touvron et al., 2023; Dubey et al., 2024; Li et al., 2023; Liu et al., 2023; Zhu et al., 2023; Bai et al., 2023b; 2025; Chu et al., 2023; 2024), natively multimodal systems have drawn substantial attention (OpenAI, 2024; Comanici et al., 2025; Xu et al., 2025).”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “Qwen-Audio is renowned for its robust performance across various audio understanding tasks, providing a strong comparison point for our model.”