data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

By Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli. 2022

1,194 citations recorded by the source corpus.

While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech, NLP or computer vision. The core idea is to predict latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which are local in nature, data2vec predicts contextualized latent representations that contain information from the entire input. Experiments on the major benchmarks of speech recognition, image classification, and natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “Examples include ResNet [33] in computer vision, BERT [19] and GPT [10, 49] in natural language, Wav2Vec [54, 3] in speech, and CLIP [48] and Data2Vec [2] in multimodal applications.”
MAESTRO: Matched Speech Text Representations through Modality Matching (2022, background): “With the convergence of architectures [14–16] and objectives [9, 17] used for learning representations in speech and natural language processing, there has been growing interest in learning joint speech and text representations.”
RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “We select the widely-used self-supervised pretrained models, HuBERT (Hsu et al., 2021) and data2vec (Baevski et al., 2022), as the speech encoder of RepCodec.”
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “Data2vec (Baevski et al., 2022) uses a fast convolutional decoder and explores the contextualized target representations in a self-supervised manner.”
Language Models are General-Purpose Interfaces (2022, background): “Baevski et al. (2022) propose a uniﬁed learning framework for different modalities but still use modality speciﬁc encoders.”
Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, positive): “Data2vec [3] employs a fast convolutional decoder and models the contextualized target representations in a self-supervised manner.”
A Survey of Visual Transformers (2021, methodology): “Combining masked modeling and contrastive learning pre-training strategy, Data2Vec [152] proposes a self-distilled network treating the masked features as a type of data augmentation, whose structure is analogous to DINO (2021) [73].”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “…the non-contrastive collapse prevention strategy in BYOL (Grill et al., 2020) was recently made to work with image feature prediction methods (Baevski et al., 2022b; Assran et al., 2023), which demonstrated the ability to learn representations that can be leveraged for various downstream…”
Masked Siamese Networks for Label-Efficient Learning (2022, methodology): “Nevertheless, masked auto-encoders have enabled the training of large-scale models and demonstrated state-of-the-art performance when ﬁne-tuning on large labeled datasets, with millions of labels (Bao et al., 2021; He et al., 2021; Xie et al., 2021; Baevski et al., 2022).”
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, background): “Recent advancements in addressing problems from a uniﬁed perspective have yielded substantial progress [16], [17], [18], [19].”
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, methodology): “Motivated by masked language modeling [32, 10], MAE [18] and some other methods [46, 53, 3] adopt asymmetric encoder-decoder transformers [13] to apply masked autoencoding for self-supervised learning on 2D images.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “ponential moving average (EMA) teacher for self-supervised visual representations [139], [140], the data2vec model [132] computes targets Y using an EMA of its own parameters.”
DINOv2: Learning Robust Visual Features without Supervision (2023, background): “Recently, the emergence of patch-based architectures, like ViTs, has led to a revisit of inpainting for pre-training (He et al., 2022; Bao et al., 2021; El-Nouby et al., 2021), potentially in feature space (Assran et al., 2023; Baevski et al., 2022).”
emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation (2023, methodology): “Representative models of this type are data2vec (Baevski et al., 2022), data2vec 2.0 (Baevski et al., 2023) using frame-level mask language model (MLM) loss, and CA-DINO (Han et al., 2023) using utterance-level cross-entropy loss. emotion2vec is pre-trained combining both utterance-level loss and…”
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (2021, background): “Most recently, numerous works (Bao, Dong, and Wei 2021; He et al. 2021; Xie et al. 2021; Chen et al. 2022; Wang et al. 2022b; Dong et al. 2022; Baevski et al. 2022; Zheng et al. 2022) based on MIM have been concurrently developed, yet few studied the perceptual level of the prediction targets.”
Masked Autoencoders that Listen (2022, methodology): “, contrastive [39, 33, 40, 41, 35] or prediction/reconstruction [18, 34, 37, 36]).”
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers (2022, background): “Knowledge distillation (Wei et al., 2021, 2022) and self-distillation (Baevski et al., 2022) proposed to mimic the features provided by the teacher at the masked positions.”