Evaluating Visual Conversational Agents via Cooperative Human-AI Games

By Prithvijit Chattopadhyay, Deshraj Yadav, Viraj Prabhu, Arjun Chandrasekaran, Abhishek Das, Stefan Lee, Dhruv Batra, Devi Parikh. 2017

80 citations recorded by the source corpus.

As AI continues to advance, human-AI teams are inevitable. However, progress in AI is routinely measured in isolation, without a human in the loop. It is crucial to benchmark progress in AI, not just in isolation, but also in terms of how it translates to helping humans perform certain tasks, i.e., the performance of human-AI teams. In this work, we design a cooperative game — GuessWhich — to measure human-AI team performance in the specific context of the AI being a visual conversational agent. GuessWhich involves live interaction between the human and the AI. The AI, which we call ALICE, is provided an image which is unseen by the human. Following a brief description of the image, the human questions ALICE about this secret image to identify it from a fixed pool of images. We measure performance of the human-ALICE team by the number of guesses it takes the human to correctly identify the secret image after a fixed number of dialog rounds with ALICE. We compare performance of the human-ALICE teams for two versions of ALICE. Our human studies suggest a counterintuitive trend – that while AI literature shows that one version outperforms the other when paired with an AI questioner bot, we find that this improvement in AI-AI performance does not translate to improved human-AI performance. This suggests a mismatch between benchmarking of AI in isolation and in the context of human-AI teams.

Subfields: Artificial Intelligence, Computer Science Applications, Dialogue & Conversational AI, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Jianfeng Gao
3. Mohit Bansal
4. Linjie Li
5. Jingjing Liu
6. Devi Parikh
7. Yu Cheng
8. Dhruv Batra

In Computer Science Applications

1. Dhruv Batra
2. Prithvijit Chattopadhyay
3. Richard Ryan Williams

In Dialogue & Conversational AI

1. Verena Rieser
2. Takayuki Okatani
3. Jianfeng Gao
4. Van-Quang Nguyen
5. Masanori Suganuma

In Human-Computer Interaction

1. Zhe Gan
2. Devi Parikh
3. Verena Rieser
4. Dhruv Batra
5. Jianfeng Gao
6. Joon-Young Lee
7. Jue Wang
8. Irfan Essa

Citations in the corpus

• VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “In addition to its scientific value, the task finds practical applications such as facilitating people with impaired vision [7, 11] and generating medical imaging reports under human supervision [47, 10].”
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “Reinforcement Learning (RL) was used in [10, 5] to train two agents to play image guessing games.”
• History for Visual Dialog: Do we really need it? (2020, background): “(Chattopadhyay et al., 2017) and GuessWhat?!”
• EvalAI: Towards Better Evaluation Systems for AI Agents (2019, background): “For instance, generating a natural language description for an image, having a conversation with a human, or generating aesthetically pleasing images cannot be evaluated accurately using automatic metrics as performance on these metrics do not correlate well with human-judgment in practice [6].”
• Audio Visual Scene-Aware Dialog (2019, background): “One attractive alternative that we leave for future work is to evaluate directly with human users in cooperative tasks [5].”
• Grounded Agreement Games: Emphasizing Conversational Grounding in Visual Dialogue Settings (2019, background): “…setting, the questioner is asking questions with the later goal in mind of identifying the image from a set also containing distractor images (Chattopadhyay et al., 2017).2 These variants of the general setting provide purpose to the questioner, but not to the answerer, which is the target…”
• Large-Scale Answerer in Questioner's Mind for Visual Dialog Question Generation (2019, background): “However, even though the generated dialog is tuned to be like human dialog, the performance of RL-QA on the conversation with human would decrease compared to SL-Q, because the distribution of Abot become far from human’s (Chattopadhyay et al., 2017; Lee et al., 2018).”
• Efficient Attention Mechanism for Handling All the Interactions between Many Inputs with Application to Visual Dialog (2019, background): “There are also other approaches that are not fit for the above categorizations, including GNN [48], which models relations in a dialog by an unknown graph structure to resolve via an EM algorithm and [4, 9], which use reinforcement learning to train two agents to play a guessing game.”