Evaluating Theory of Mind in Question Answering

By Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, T. Griffiths. 2018

110 citations recorded by the source corpus.

We propose a new dataset for evaluating question answering models with respect to their capacity to reason about beliefs. Our tasks are inspired by theory-of-mind experiments that examine whether children are able to reason about the beliefs of others, in particular when those beliefs differ from reality. We evaluate a number of recent neural models with memory augmentation. We find that all fail on our tasks, which require keeping track of inconsistent states of the world; moreover, the models’ accuracy decreases notably when random sentences are introduced to the tasks at test.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Question Answering & Reading Comprehension

Citations in the corpus

Experience Grounds Language (2020, background): “The ability to consider the feelings and knowledge of others is now commonly referred to as the “Theory of Mind” (Nematzadeh et al., 2018).”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, background): “Theory of Mind Guanaco displays surprisingly strong Theory of Mind capabilities [41, 52].”
Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, methodology): “Inspired by false-belief ToM evaluation protocols in cognitive science, Nematzadeh et al. (2018) and Le et al. (2019) have proposed ﬁrst-order and second-order belief evaluation tasks for language models (also see Section Dysfunction of Theory of Mind for discussion about the interaction of ToM and…”
Revisiting the Evaluation of Theory of Mind through Question Answering (2019, background): “Perner and Wimmer (1985) proposed a set of experiments to test such second-order beliefs in children.”
ALERT: Adapt Language Models to Reasoning Tasks (2022, background): “, 2020) tomqa find location easy clean (Nematzadeh et al., 2018)”
Few-shot Language Coordination by Modeling Theory of Mind (2021, background): “Rabinowitz et al. (2018) are the first to present a successful modeling of the mental state of various species of agent.”