AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

By Madeleine Grunde-McLaughlin, Ranjay Krishna, Maneesh Agrawala. 2021

172 citations recorded by the source corpus.

Visual events are a composition of temporal actions involving actors spatially interacting with objects. When developing computer vision models that can reason about compositional spatio-temporal events, we need benchmarks that can analyze progress and uncover shortcomings. Existing video question answering benchmarks are useful, but they often conflate multiple sources of error into one accuracy metric and have strong biases that models can exploit, making it difficult to pinpoint model weaknesses. We present Action Genome Question Answering (AGQA), a new benchmark for compositional spatio-temporal reasoning. AGQA contains 192M unbalanced question answer pairs for 9.6K videos. We also provide a balanced subset of 3.9M question answer pairs, 3 orders of magnitude larger than existing benchmarks, that minimizes bias by balancing the answer distributions and types of question structures. Although human evaluators marked 86.02% of our question-answer pairs as correct, the best model achieves only 47.74% accuracy. In addition, AGQA introduces multiple training/test splits to test for various reasoning abilities, including generalization to novel compositions, to indirect references, and to more compositional steps. Using AGQA, we evaluate modern visual reasoning systems, demonstrating that the best models barely perform better than non-visual baselines exploiting linguistic biases and that none of the existing models generalize to novel compositions unseen during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Li
2. Noah A. Smith
3. Mohit Bansal
4. Percy Liang
5. Jungo Kasai
6. Ranjay Krishna
7. Yejin Choi
8. Yizhong Wang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Bo Li
3. Jitendra Malik
4. Xiaogang Wang
5. Ming-Hsuan Yang
6. Dacheng Tao
7. Trevor Darrell
8. Li Fei-Fei

In Evaluation & Benchmarks

1. Noah A. Smith
2. Jungo Kasai
3. Jitendra Malik
4. Bo Li
5. Mari Ostendorf
6. Ranjay Krishna
7. Yizhong Wang
8. Yushi Hu

Citations in the corpus

• STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Visual Reasoning Beyond visual question answering, several new datasets [21, 18, 47, 14, 16, 6] are designed to diagnose models’ reasoning abilities.”
• EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[21] Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala.”
• Long Context Transfer from Language to Vision (2024, background): “These range from fundamental visual perception tasks such as activity recognition[84], concept detection [80], and counting [28], to more complex visual reasoning tasks including compositional [24], causal [76, 83, 81], and situated reasoning [74].”
• TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering (2023, result): “When there are more than 5 entities, The TIFA score starts to drop rapidly for all text-to-image models, consistent with similar findings in other vision-language evaluations [15, 13].”