A Closer Look at the Robustness of Vision-and-Language Pre-trained Models

By Linjie Li, Zhe Gan, Jingjing Liu. 2020

50 citations recorded by the source corpus.

Large-scale pre-trained multimodal transformers, such as ViLBERT and UNITER, have propelled the state of the art in vision-and-language (V+L) research to a new level. Although achieving impressive performance on standard tasks, to date, it still remains unclear how robust these pre-trained models are. To investigate, we conduct a host of thorough evaluations on existing pre-trained models over 4 different types of V+L specific model robustness: (i) Linguistic Variation; (ii) Logical Reasoning; (iii) Visual Content Manipulation; and (iv) Answer Distribution Shift. Interestingly, by standard model finetuning, pre-trained V+L models already exhibit better robustness than many task-specific state-of-the-art methods. To further enhance model robustness, we propose Mango, a generic and efficient approach that learns a Multimodal Adversarial Noise GeneratOr in the embedding space to fool pre-trained V+L models. Differing from previous studies focused on one specific type of robustness, Mango is task-agnostic, and enables universal performance lift for pre-trained models over diverse tasks designed to evaluate broad aspects of robustness. Comprehensive experiments demonstrate that Mango achieves new state of the art on 7 out of 9 robustness benchmarks, surpassing existing methods by a significant margin. As the first comprehensive study on V+L robustness, this work puts robustness of pre-trained models into sharper focus, pointing new directions for future study.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Xiaowei Hu
  6. Zhengyuan Yang
  7. Jianfeng Gao
  8. Douwe Kiela

In Computer Vision and Pattern Recognition

  1. Zhe Gan
  2. Lijuan Wang
  3. Zicheng Liu
  4. Jianfeng Wang
  5. Xiaowei Hu
  6. Zhengyuan Yang
  7. Jianfeng Gao
  8. Cordelia Schmid

In Multimodal Learning

  1. Lijuan Wang
  2. Jianfeng Wang
  3. Zicheng Liu
  4. Zhe Gan
  5. Zhengyuan Yang
  6. Xiaowei Hu
  7. Yumao Lu
  8. Yezhou Yang

In Safety, Robustness & Interpretability

  1. Adina Williams
  2. Douwe Kiela
  3. Tristan Thrush
  4. Candace Ross
  5. Zhe Gan
  6. Max Bartolo
  7. Yezhou Yang
  8. Eric Michael Smith

Citations in the corpus