BEATs: Audio Pre-Training with Acoustic Tokenizers

By Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel C. Tompkins, Zhuo Chen, Furu Wei. International Conference on Machine Learning, 2022

641 citations recorded by the source corpus.

The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/89a1dbbfd4c96d90b769f5d3427bd970b082898e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Limin Wang
2. Yu Qiao
3. Yali Wang
4. Kunchang Li
5. Yi Wang
6. Yinan He
7. Jiashuo Yu
8. Rita Singh

In Signal Processing

1. Rita Singh
2. Huaming Wang
3. Soham Deshmukh
4. Yu Qiao
5. Guangzhi Sun
6. Limin Wang
7. Benjamin Elizalde
8. Yi Wang

Citations in the corpus

• InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “VALOR [Chen et al., 2023b] employed independent encoders for video, audio, and text and trains a joint visual-audio-text representation.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “❶ The first principle guides us to avoid using resampler architecture (Li et al., 2023a; Alayrac et al., 2022) because the resampling operation does not guarantee the preservation of spatial-temporal order.”
• SALMONN: Towards Generic Hearing Abilities for Large Language Models (2023, methodology): “The tokeniser is updated by distilling the semantic knowledge of the audio tokens (Chen et al., 2023c).”
• Pengi: An Audio Language Model for Audio Tasks (2023, background): “These methods do not require labels [55, 7] and have achieved state-of-the-art performance.”