MultiMAE: Multi-modal Multi-task Masked Autoencoders

By Roman Bachmann, David Mizrahi, Andrei Atanov, Amir Zamir. European Conference on Computer Vision, 2022

412 citations recorded by the source corpus.

We propose a pre-training strategy called Multi-modal Multi-task Masked Autoencoders (MultiMAE). It differs from standard Masked Autoencoding in two key aspects: I) it can optionally accept additional modalities of information in the input besides the RGB image (hence"multi-modal"), and II) its training objective accordingly includes predicting multiple outputs besides the RGB image (hence"multi-task"). We make use of masking (across image patches and input modalities) to make training MultiMAE tractable as well as to ensure cross-modality predictive coding is indeed learned by the network. We show this pre-training strategy leads to a flexible, simple, and efficient framework with improved transfer results to downstream tasks. In particular, the same exact pre-trained network can be flexibly used when additional information besides RGB images is available or when no information other than RGB is available - in all configurations yielding competitive to or significantly better results than the baselines. To avoid needing training datasets with multiple modalities and tasks, we train MultiMAE entirely using pseudo labeling, which makes the framework widely applicable to any RGB dataset. The experiments are performed on multiple transfer tasks (image classification, semantic segmentation, depth estimation) and datasets (ImageNet, ADE20K, Taskonomy, Hypersim, NYUv2). The results show an intriguingly impressive capability by the model in cross-modal/task predictive coding and transfer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b1fc7d96d732f99510658c73a8d9da3fd7b25923

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ImageBind One Embedding Space to Bind Them All (2023, methodology): “For few-shot depth classification, we compare with the multimodal MultiMAE [4] ViT-B/16 model trained on images, depth, and semantic segmentation data.”
Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “Simple and free from dataset or task-specific augmentations [43], MAE is the state95 of-the-art self-supervised framework in computer vision [44, 45, 46, 47], and has been demonstrated 96 to work well for motor control tasks in simulation [9].”
Scaling Language-Image Pre-Training via Masking (2022, methodology): “The MAE design has been applied to videos [61, 22], point clouds [49], graphs [59, 9, 32], audio [4, 47, 13, 35], visual control [70, 57], vision-language [23, 41, 31, 19], and other modalities [5].”
SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022, background): “MultiMAE [48] takes optional inputs of different modalities and accordingly includes other training objectives to facilitate multi-modality learning.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “Beyond size, some works have also investigated domain issues in data and found that they can be alleviated by training the masked autoencoder on images of mixed-style [65] or multi-tasks [66].”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, background): “Regarding the optimization of MAE and multimodal learning (MML) that may contradict each other, combining them without compromising their merits remains an open question [29].”
Images Speak in Images: A Generalist Painter for In-Context Visual Learning (2022, background): “In NLP, the emergence of in-context learning [2, 7, 18] presents a new path in this direction, which uses language sequences as the general interface and allows the model to rapidly adapt to various language-centric tasks with only a handful of prompts and examples.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, background): “Moreover, there are multi-task frameworks [71], [72], [73] that facilitate inter-task feature propagation for RGB-D scene understanding, but they rely on supervision from”