Omnivore: A Single Model for Many Visual Modalities

By Rohit Girdhar, Mannat Singh, Nikhil Ravi, L. V. D. van der Maaten, Armand Joulin, Ishan Misra. 2022

319 citations recorded by the source corpus.

Prior work has studied different visual modalities in isolation and developed separate architectures for recognition of images, videos, and 3D data. Instead, in this paper, we propose a single model which excels at classifying images, videos, and single-view 3D data using exactly the same model parameters. Our ‘OMNIVORE’ model leverages the flexibility of transformer-based architectures and is trained jointly on classification tasks from different modalities. Omnivoreis simple to train, uses off-the-shelf standard datasets, and performs at-par or better than modality-specific models of the same size. A single Omnivoremodel obtains 86.0% on ImageNet, 84.1% on Kinetics, and 67.1% on SUN RGB-D. After finetuning, our models outperform prior work on a variety of vision tasks and generalize across modalities. OMNIVORE's shared visual representation naturally enables cross-modal recognition without access to correspondences between modalities. We hope our results motivate researchers to model visual modalities together.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

SOCRATES: Introducing Depth in Visual Wildlife Monitoring Using Stereo Vision (2022, methodology): “In our experiments, we use Omnivore as our backbone of choice.”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “, a Transformer [68]) to operate on modalities such as images and text [3, 11, 15, 29, 31–34, 41, 42, 59, 62, 75], video and audio [4, 30, 44, 46], video, text and audio [2], and depth, images and video [27].”
Multimodal Learning With Transformers: A Survey (2022, background): “A large amount of recent attempts [117], [296], [297], [298] study how to use as unified as possible pipelines to handle various modalities and multimodal tasks.”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, unknown): “3 OMNIVORE(Swin-L)[21] IN22K+SUN 32×3×4 7248 84.”
ImageBind One Embedding Space to Bind Them All (2023, methodology): “Prior works explored joint training of multiple modalities in a supervised [20, 41] or self-supervised contexts [3, 19, 49, 68, 72].”