On Modality Bias in the TVQA Dataset

By Thomas Winterbottom, Sarah Xiao, Alistair McLean, N. A. Moubayed. 2020

49 citations recorded by the source corpus.

TVQA is a large scale video question answering (video-QA) dataset based on popular TV shows. The questions were specifically designed to require"both vision and language understanding to answer". In this work, we demonstrate an inherent bias in the dataset towards the textual subtitle modality. We infer said bias both directly and indirectly, notably finding that models trained with subtitles learn, on-average, to suppress video feature contribution. Our results demonstrate that models trained on only the visual information can answer ~45% of the questions, while using only the subtitles achieves ~68%. We find that a bilinear pooling based joint representation of modalities damages model performance by 9% implying a reliance on modality specific information. We also show that TVQA fails to benefit from the RUBi modality bias reduction technique popularised in VQA. By simply improving text processing using BERT embeddings with the simple model first proposed for TVQA, we achieve state-of-the-art results (72.13%) compared to the highly complex STAGE model (70.50%). We recommend a multimodal evaluation framework that can highlight biases in models and isolate visual and textual reliant subsets of data. Using this framework we propose subsets of TVQA that respond exclusively to either or both modalities in order to facilitate multimodal modelling as TVQA originally intended.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Angela Yao
2. Tat-seng Chua
3. Yake Wei
4. Andong Deng
5. Xiaokang Peng
6. Di Hu
7. Xindi Shang
8. Dong Wang

In Computer Vision and Pattern Recognition

1. Tat-seng Chua
2. Junbin Xiao
3. Angela Yao
4. Xindi Shang
5. Yake Wei
6. Andong Deng
7. Xiaokang Peng
8. Di Hu

In Multimodal Learning

1. Angela Yao
2. Tat-seng Chua
3. Yake Wei
4. Andong Deng
5. Xiaokang Peng
6. Di Hu
7. Xindi Shang
8. Dong Wang

In Evaluation & Benchmarks

1. Angela Yao
2. Tat-seng Chua
3. Xindi Shang
4. Junbin Xiao

Citations in the corpus

• NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “Nonetheless, they are either biased to textual plot understanding or actor dialogue comprehension [47], severely diminishing their challenge for visual reasoning.”
• Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “In contrast to some previous works using additional video features such as subtitles [12, 36, 37, 43, 44, 46, 74, 80, 87], our video representation is exclusively based on visual information, as we focus on the visual understanding of videos.”
• Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, background): “Several studies pointed out that most multimodal learning methods cannot effectively boost the performance even with more information [8,35,39,40], which is caused by the discrepancy between modalities.”