Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence

By Amir Zadeh, Michael Chan, Percy Liang, Edmund Tong, Louis-philippe Morency. 2019

179 citations recorded by the source corpus.

As intelligent systems increasingly blend into our everyday life, artificial social intelligence becomes a prominent area of research. Intelligent systems must be socially intelligent in order to comprehend human intents and maintain a rich level of interaction with humans. Human language offers a unique unconstrained approach to probe through questions and reason through answers about social situations. This unconstrained approach extends previous attempts to model social intelligence through numeric supervision (e.g. sentiment and emotions labels). In this paper, we introduce Social-IQ, a unconstrained benchmark specifically designed to train and evaluate socially intelligent technologies. By providing a rich source of open-ended questions and answers, Social-IQ opens the door to explainable social intelligence. The dataset contains rigorously annotated and validated videos, questions and answers, as well as annotations for the complexity level of each question and answer. Social-IQ contains 1,250 natural in-the-wild social situations, 7,500 questions and 52,500 correct and incorrect answers. Although humans can reason about social situations with very high accuracy (95.08%), existing state-of-the-art computational models struggle on this task. As a result, Social-IQ brings novel challenges that will spark future research in social intelligence modeling, visual reasoning, and multimodal question answering (QA).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Experience Grounds Language (2020, background): “These activities require inferring mental states and social outcomes—a key area of interest in itself (Zadeh et al., 2019).”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “We encourage further research in developing benchmarks that measure specific social abilities in LLMs (e.g., Sap et al., 2019b; Zadeh et al., 2019), especially those that minimize annotation artifacts and spurious correlations (Schwartz et al.”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, neutral): “As result, current VideoQA datasets [12, 17, 18, 22, 28, 35, 40, 45, 48, 62, 71, 77, 87, 93, 94, 95, 96] are limited in size, as the largest, TGIF-QA [35], contains only 72K annotated clips (see Figure 2 for more details).”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “The VCR dataset (Zellers (Tapaswi et al., 2016) × × × MarioQA (Mun et al., 2017) × × × TVQA (Lei et al., 2018) × × × × Social-IQ (Zadeh et al., 2019) × × × ×”
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents (2023, background): “On one hand, many of the social intelligence benchmarks are not interactive (Sap et al., 2019; Le et al., 2019; Zadeh et al., 2019b), which is sub-optimal for evaluating social intelligence ((Mehri et al., 2022; Hoppler et al., 2022; Lee et al., 2023)).”
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “Similarly, [18] and [35] propose videobased QA tasks with open-ended high-order questions that need multi-modal understanding, social intelligence modeling, and visual reasoning.”
Video Instruction Tuning With Synthetic Data (2024, unknown): “…manually annotated data for various tasks, such as video captions (Chen & Dolan, 2011; Xu et al., 2016; Rohrbach et al., 2015; Anne Hendricks et al., 2017a; Caba Heilbron et al., 2015; Zhou & Corso, 2017), and video question-answering (Yu et al., 2019; Zadeh et al., 2019; Xiao et al., 2021).”
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, background): “Some prior work has explored commonsense reasoning tasks posed over videos, which usually have more input available than a single frame [12,13,19,20,30,31,33,63,74,78] (inter alia).”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “Social-IQ [38] 1250 7500 Manual YouTube Videos Multiple-choice MSRVTT-QA [7] 10,000 243,680 Automatic MSRVTT(Web videos) Open-ended”
Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning (2021, methodology): “Various methods [1, 45, 55, 53] have been proposed for visual question answering.”
What Is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020, background): “Recently, commonsense reasoning has emerged as an important topic in both language (Zellers et al., 2018, 2019b; Sap et al., 2019) and vision (Vedantam et al., 2015b; Zellers et al., 2019a; Zadeh et al., 2019; Fang et al., 2020) communities.”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, background): “Several of these prominent benchmarks rely on dialogue and plot summaries instead of a video’s visual contents [33, 48, 27, 59], resulting in models with a stronger dependence on the dialogue than on the visual input and therefore reducing the benchmark’s effectiveness at measuring visual reasoning ”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Many video question answering datasets [25, 47, 40, 65, 79] and the methods [80, 49, 36, 73, 76, 20] built on them mainly focus on understanding diverse visual scenes, such as human actions (MovieQA [65]) or 3D object movements without physical and language cues (CATER [25]).”
LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, unknown): “…manually annotated data for various tasks, such as video captions (Chen & Dolan, 2011; Xu et al., 2016; Rohrbach et al., 2015; Anne Hendricks et al., 2017a; Caba Heilbron et al., 2015; Zhou & Corso, 2017), and video question-answering (Yu et al., 2019; Zadeh et al., 2019; Xiao et al., 2021).”
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA (2020, background): “…answering on a single image, attention to understanding and extracting information from a sequence of images, i.e., a video, is rising (Tapaswi et al., 2016; Maharaj et al., 2017; Kim et al., 2017; Jang et al., 2017; Lei et al., 2018; Zadeh et al., 2019; Lei et al., 2020; Garcia et al., 2020).”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “Hence, many smaller datasets have been proposed for evaluating different aspects of video-language understanding such as compositional reasoning [21, 22], causal and common scene comprehension [52], instruction understanding [33, 56], video description ability [54], dynamic environments understandin”