Masked Autoencoders that Listen

By Po-Yao (Bernie) Huang, Hu Xu, Juncheng Billy Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, Christoph Feichtenhofer. Neural Information Processing Systems, 2022

474 citations recorded by the source corpus.

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/794c5d5ca20e71eae416da91cf1fed0a8ef15658

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

DINOv2: Learning Robust Visual Features without Supervision (2023, background): “, 2022), audio (Xu et al., 2022), and across other modalities (Girdhar et al.”
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition (2023, methodology): “Impressively, UniRepLKNet outperforms InternImage and the other models.”
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, methodology): “In our work, we adopt AudioMAE [21] and variational autoencoder (VAE) [57] as the semantic and acoustic representation learning modules, respectively.”
BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, methodology): “We keep the model size similar to the previous SOTA audio pre-trained models (Xu et al., 2022; Chong et al., 2022) for a fair comparison of the pre-training methods.”
Token Merging: Your ViT But Faster (2022, methodology): “, 2022) that can put up state-of-the art results while being fast to train; given their lack of assumptions about the data, they can be applied with little or no changes across many modalities (Feichtenhofer et al., 2022; Huang et al., 2022); and they scale well with massive amounts of data (Zhai et”
Scaling Language-Image Pre-Training via Masking (2022, methodology): “The MAE design has been applied to videos [61, 22], point clouds [49], graphs [59, 9, 32], audio [4, 47, 13, 35], visual control [70, 57], vision-language [23, 41, 31, 19], and other modalities [5].”
AudioGen: Textually Guided Audio Generation (2022, background): “More recently, Huang et al. (2022) proposed a Masked Auto Encoding approach for learning a speech representation and show it efﬁciency on several audio classiﬁcation tasks.”
Pengi: An Audio Language Model for Audio Tasks (2023, methodology): “In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques [52, 53, 45, 24, 27, 26, 3, 17, 28, 25, 59, 60, 43, 13, 16, 18, 65, 10].”
ImageBind One Embedding Space to Bind Them All (2023, unknown): “IMAGEBIND significantly outperforms the AudioMAE model on all settings with gains of ∼40% accuracy in top-1 accuracy on ≤4-shot classification.”
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (2023, methodology): “Moreover, MAE pretraining is sparse and can be 4 − 10 × as fast as normal supervised training, making it an already desirable alternative across many domains for more than just accuracy (He et al., 2022; Feichtenhofer et al., 2022; Huang et al., 2022b).”