Masked Feature Prediction for Self-Supervised Visual Pre-Training

By Chen Wei, Haoqi Fan, Saining Xie, Chaoxia Wu, Alan L. Yuille, Christoph Feichtenhofer. Computer Vision and Pattern Recognition, 2021

867 citations recorded by the source corpus.

We present Masked Feature Prediction (MaskFeat) for self-supervised pre-training of video models. Our approach first randomly masks out a portion of the input sequence and then predicts the feature of the masked regions. We study five different types of features and find Histograms of Oriented Gradients (HOG), a hand-crafted feature descriptor, works particularly well in terms of both performance and efficiency. We observe that the local contrast normalization in HOG is essential for good results, which is in line with earlier work using HOG for visual recognition. Our approach can learn abundant visual knowledge and drive large-scale Transformer based models. Without using extra model weights or supervision, MaskFeat pretrained on unlabeled videos achieves unprecedented results of 86.7% with MViTv2-L on Kinetics-400, 88.3% on Kinetics 600, 80.4% on Kinetics-700, 38.8 mAP on AVA, and 75.0% on SSv2. MaskFeat further generalizes to image input, which can be interpreted as a video with a single frame and obtains competitive results on ImageN et.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/008a428e049003fe768068a0f1fa1416af5c4982

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “BeiT [20] propose visual token prediction with the pretrained tokenizer [44], MaskFeat [6] predicts the hand-crafted image descriptor, and MAE [25] directly reconstructs the raw pixels.”
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers (2022, background): “Knowledge distillation (Wei et al., 2021, 2022) and self-distillation (Baevski et al., 2022) proposed to mimic the features provided by the teacher at the masked positions.”
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (2021, background): “Current state-of-the-art self-supervised pre-training methods (Dosovitskiy et al. 2020; Bao, Dong, and Wei 2021; He et al. 2021; Xie et al. 2021; Chen et al. 2022; Wei et al. 2021) for vision transformers focus on masked image modeling (MIM), a task of making predictions for masked patches from the ”
Masked Autoencoders for Point Cloud Self-supervised Learning (2022, methodology): “These methods have dominated until recent generative SSL methods [17,49,45] result in more competitive performance.”
Context Autoencoder for Self-supervised Representation Learning (2022, neutral): “, 2021), Masked Feature Prediction (MaskFeat) (Wei et al., 2021), Simple MIM (SimMIM) (Xie et al.”
RoMa: Robust Dense Feature Matching (2023, positive): “A recent promising direction for frozen pretrained features is large-scale self-supervised pretraining using Masked image Modeling (MIM) [24, 37, 56, 62].”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “Further, different training methods have also been explored, e.g. , unsupervised learning [74, 23, 84], and video-text contrastive learning [71, 87, 86, 73].”
MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation (2022, methodology): “Given a partly masked image, the network is trained to reconstruct properties of the masked areas such as VAE features [2, 17, 51], HOG features [93], or color information [29, 99].”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, background): “…and motivated by the success of BERT [24] in NLP, many recent works propose a variety of masked image prediction methods for pre-training vision models in a self-supervised way, using reconstruction targets such as pixels [6, 16, 26, 29, 35, 93], discrete tokens [9,103], and (deep) features [8,88].”
Scaling Language-Image Pre-Training via Masking (2022, background): “In computer vision, explorations along this direction include predicting large missing regions [50], sequence of pixels [10], patches [20, 29, 71], or pre-computed features [6, 66].”
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (2023, background): “This idea is at the core of self-supervised generative methods, which remove or corrupt portions of the input and learn to predict the corrupted content [8, 35, 56, 65, 66, 69].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Low-Level Targets High-Level Targets Self-Distillation Contrastive / Multi-modal Teacher Algorithm ViT [5] MAE [68] SimMIM [114] Maskfeat [118] BEiT [112] CAE [113] PeCo [119] data2vec [120] SdAE [121] MimCo [122] BEiT v2 [123] Target Raw Pixel HOG VQ-VAE VQ-GAN self MoCo v3 CLIP”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “, 2020b) or BEiT, MAE, SimMIM, and MaskFeat (Bao et al., 2021; He et al., 2021; Xie et al., 2021; Wei et al., 2021) which predict targets lacking contextual information.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “As a concurrent work, MaskFeat [152] mainly studies reconstructing targets of the MIM framework, such as the histogram of oriented gradient (HOG) features.”
Masked Autoencoders As Spatiotemporal Learners (2022, methodology): “: Space-only random sampling, broadcasted to all time steps (“tube” masking [77]).”
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, background): “The follow-up works further improve the performance of MAE by momentum encoder [53], contrastive learning [3], and modified reconstruction targets [42].”