MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
By Yizhi Li, Ruibin Yuan, Ge Zhang, Yi Ma, Xingran Chen, Hanzhi Yin, Chen-Li Lin, Anton Ragni, Emmanouil Benetos, N. Gyenge and 8 more. International Conference on Learning Representations, 2023
320 citations recorded by the source corpus.
Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is partially due to the distinctive challenges associated with modelling musical knowledge, particularly tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified an effective combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantisation - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing
View Paper: https://www.semanticscholar.org/paper/fb08b08916caab4b22c60fa96753f6b9a5886d75
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Signal Processing
Citations in the corpus
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, unknown): “This model is based on MERT (Li et al., 2023) and Llama (Touvron et al., 2023b) (xi) Gemini-Flash and Gemini-Pro (Team et al., 2024): Two proprietary LALMs known for advanced capabilities in speech, music, and sound reasoning.”
- SALMONN: Towards Generic Hearing Abilities for Large Language Models (2023, methodology): “Unlike speech, audio event inputs are often treated as fixed-sized spectrogram images that can be processed using visual-language LLM methods without explicitly modelling temporal correlations (Gong et al., 2023a;b; Zhang et al., 2023b).”