NLVR2 Visual Bias Analysis

By Alane Suhr, Yoav Artzi. 2019

23 citations recorded by the source corpus.

NLVR2 (Suhr et al., 2019) was designed to be robust for language bias through a data collection process that resulted in each natural language sentence appearing with both true and false labels. The process did not provide a similar measure of control for visual bias. This technical report analyzes the potential for visual bias in NLVR2. We show that some amount of visual bias likely exists. Finally, we identify a subset of the test data that allows to test for model performance in a way that is robust to such potential biases. We show that the performance of existing models (Li et al., 2019; Tan and Bansal 2019) is relatively robust to this potential bias. We propose to add the evaluation on this subset of the data to the NLVR2 evaluation protocol, and update the official release to include it. A notebook including an implementation of the code used to replicate this analysis is available at this http URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Hannaneh Hajishirzi
3. Matt Gardner
4. Yejin Choi
5. Zhe Gan
6. Jonathan Berant
7. Jianfeng Gao
8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Hannaneh Hajishirzi
3. Mohit Bansal
4. Jingjing Liu
5. Linjie Li
6. Jianfeng Gao
7. Luc Van Gool
8. Dacheng Tao

In Multimodal Learning

1. Zhe Gan
2. Linjie Li
3. Jingjing Liu
4. Yu Cheng
5. Yen-Chun Chen
6. Lijuan Wang
7. Mohit Bansal
8. Jianfeng Gao

In Evaluation & Benchmarks

1. Matt Gardner
2. Daniel Khashabi
3. Hannaneh Hajishirzi
4. Jonathan Berant
5. Sameer Singh
6. Ashish Sabharwal
7. Eric Wallace
8. Noah A. Smith

Citations in the corpus

• Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “We constructed NLVR2 contrast sets by modifying the sentence or replacing one of the images with freely-licensed images from web searches.”
• Evaluating NLP Models via Contrast Sets (2020, methodology): “We call this metric contrast consistency, following prior work with related evaluation metrics (Goldman et al., 2018; Suhr and Artzi, 2019).”
• UNITER: UNiversal Image-TExt Representation Learning (2019, background): “6 The balanced and unbalanced evaluations were introduced in [17].”