CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text

By Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton. 2019

274 citations recorded by the source corpus.

The recent success of natural language understanding (NLU) systems has been troubled by results highlighting the failure of these models to generalize in a systematic and robust way. In this work, we introduce a diagnostic benchmark suite, named CLUTRR, to clarify some key issues related to the robustness and systematicity of NLU systems. Motivated by the classic work on inductive logic programming, CLUTRR requires that an NLU system infer kinship relations between characters in short stories. Successful performance on this task requires both extracting relationships between entities, as well as inferring the logical rules governing these relationships. CLUTRR allows us to precisely measure a model’s ability for systematic generalization by evaluating on held-out combinations of logical rules, and allows us to evaluate a model’s robustness by adding curated noise facts. Our empirical results highlight a substantial performance gap between state-of-the-art NLU models (e.g., BERT and MAC) and a graph neural network model that works directly with symbolic inputs—with the graph-based model exhibiting both stronger generalization and greater robustness.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Transformers as Soft Reasoners over Language (2020, methodology): “To generate our data, we follow a similar approach to [ Sinha et al. , 2019 ] , described below.”
Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2 (2021, methodology): “Likewise, Gontier et al. [2020] demonstrate that transformer language models can be trained to generate effective context expansions that allow the model to solve reasoning tasks from CLUTTR, a database of inference problems with implicit general premises [Sinha et al., 2019].”
Critical Thinking for Language Models (2020, background): “CLUTRR is a task generator for relational reasoning on kinship graphs (Sinha et al., 2019).”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “…et al., 2020) (Saha et al., 2020) Binary Classification, Sequence Generation 2020 PRover [RoBERTa- based](Encoder) No NA RuleTaker (Sinha et al., 2019) (Sinha et al., 2019) Sequence Generation 2019 BERT (Encoder) No NA CLUTRR • RoBERTa: In a 2020 study by Clark et al. (2020b), a…”
Competition-Level Problems are Effective LLM Evaluators (2023, background): “…et al., 2021), numerical reasoning (Dua et al., 2019), multi-hop reasoning (Yang et al., 2018), arithmetic reasoning (Patel et al., 2021; Cobbe et al., 2021), structured reasoning (Yu et al., 2018; Lei et al., 2023), inductive reasoning (Sinha et al., 2019) and logical reasoning (Yu et al., 2020).”
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, unknown): “An interesting finding along with CLUTRR was that Chat-GPT can’t differentiate son and grandson but can differentiate daughter and granddaughter when it induces the logical rules governing kinship relationships.”
On the Advance of Making Language Models Better Reasoners (2022, methodology): “…2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al., 2019) and tabular reasoning (Chen et al., 2020; Zhu et al., 2021).”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, background): “There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions (Weston et al., 2015; Sinha et al., 2019; Clark et al., 2021; Han et al., 2022; Tafjord et al., 2020, inter alia ).”
Better Set Representations For Relational Reasoning (2020, methodology): “Finally, we show that our approach is also applicable over inputs other than images through the text-based relational reasoning task CLUTRR [16].”
Making Language Models Better Reasoners with Step-Aware Verifier (2022, methodology): “…2018), arithmetic reasoning (Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Miao et al., 2020; Patel et al., 2021; Cobbe et al., 2021), logical reasoning (Liu et al., 2020; Yu et al., 2020), inductive reasoning (Sinha et al., 2019) and tabular reasoning (Chen et al., 2020; Zhu et al., 2021).”
On the Paradox of Learning to Reason from Data (2022, methodology): “However, most studies focus on solving a single task, and the datasets either are designed for a speciﬁc domain (Johnson et al., 2017; Sinha et al., 2019), or have confounding factors such as language variance (Yu et al., 2020); they can not be used to strictly or comprehensively study the logical…”
Reasoning with Language Model Prompting: A Survey (2022, background): “As opposed to deductive reasoning, inductive reasoning aims to draw conclusions by going from specific observations to general principles (Sinha et al., 2019; Yang et al., 2022c).”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, background): “Several other benchmarks in this vein also show bad performance: ReClor [Yu et al., 2020], QuAIL [Rogers et al., 2020a], ConTRoL [Liu et al., 2020a], Strate-6 gyQA [Geva et al., 2021], AR-LSAT [Zhong et al., 2021], and CLUTRR [Sinha et al., 2019].”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, methodology): “Instead, we used a BART model (Lewis et al., 2019) that was ﬁne-tuned on the CLUTRR training corpus.”
Faithful Chain-of-Thought Reasoning (2023, methodology): “We use the CLUTRR (Sinha et al., 2019) benchmark described in Section 3.4.”
Are Pretrained Language Models Symbolic Reasoners over Knowledge? (2020, background): “For complex reasoning QA benchmarks (Yang et al., 2018; Sinha et al., 2019), PLMs are ﬁnetuned to the downstream tasks.”
The Impact of Positional Encoding on Length Generalization in Transformers (2023, background): “A group of studies showed the generalization failure of conventional Transformers with APE on specific datasets such as PCFG (Hupkes et al., 2020), LEGO (Zhang et al., 2023), or CLUTRR (Sinha et al., 2019; Gontier et al., 2020).”
Learning to Solve Complex Tasks by Talking to Agents (2021, background): “…reasoning datasets as discussed above, there has been renewed interest in developing hard synthetic reasoning challenges (Lake and Baroni, 2018; Sinha et al., 2019; Ruis et al., 2020; Clark et al., 2020; Betz and Richard-son, 2021) that help to more systematically identify model strengths and…”
Graph Representation Learning (2020, background): “This approach is simple and general, and has seen recent success with attentionbased approaches as the base aggregation function [Sinha et al., 2019].”