OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

By Jing Liu, Xinxin Zhu, Fei Liu, Longteng Guo, Zijia Zhao, Ming-Ting Sun, Weining Wang, Hanqing Lu, Shiyu Zhou, Jiajun Zhang and 1 more. 2021

41 citations recorded by the source corpus.

In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructed in an encoder-decoder framework, including three single-modal encoders to generate token-based embeddings for each modality, a cross-modal encoder to encode the correlations among the three modalities, and two cross-modal decoders to generate text and image respectively. For the OPT's pre-training, we design a multi-task pretext learning scheme to model multi-modal resources from three different data granularities, \ie, token-, modality-, and sample-level modeling, through which OPT learns to align and translate among different modalities. The pre-training task is carried out on a large amount of image-text-audio triplets from Open Images. Experimental results show that OPT can learn strong image-text-audio multi-modal representations and achieve promising results on a variety of cross-modal understanding and generation tasks.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yejin Choi
2. Yi Yang
3. Ali Farhadi
4. Rowan Zellers
5. Luc Van Gool
6. Dacheng Tao
7. Mohit Bansal
8. Trevor Darrell

In Multimodal Learning

1. Ali Farhadi
2. Rowan Zellers
3. Yejin Choi
4. Jiasen Lu
5. Jack Hessel
6. Youngjae Yu
7. Xiuyi Chen
8. Ximing Lu

In Representation & Self-Supervised Learning

1. Ali Farhadi
2. Yejin Choi
3. Guangyao Chen
4. Bo Xu
5. Shuang Xu
6. Xiao Wang
7. Xiaoyong Wei
8. Rowan Zellers

Citations in the corpus

• MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[78] Jing Liu, Xinxin Zhu, Fei Liu, Longteng Guo, Zijia Zhao, Mingzhen Sun, Weining Wang, Jinqiao Wang, and Hanqing Lu.”
• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “, Oscar [17] and ALBEF [130]) and vision-languageaudio pre-training model OPT [22].”
• VLP: A Survey on Vision-language Pre-training (2022, methodology): “Differing from VATT, OPT [141] learns the cross-modal representations across text, image, and audio jointly with various multi-level masking strategies, and it is also capable of generating text and images.”