Benefits of Intermediate Annotations in Reading Comprehension

By Dheeru Dua, Sameer Singh, Matt Gardner. 2020

34 citations recorded by the source corpus.

Complex compositional reading comprehension datasets require performing latent sequential decisions that are learned via supervision from the final answer. A large combinatorial space of possible decision paths that result in the same answer, compounded by the lack of intermediate supervision to help choose the right path, makes the learning particularly hard for this task. In this work, we study the benefits of collecting intermediate reasoning supervision along with the answer during data collection. We find that these intermediate annotations can provide two-fold benefits. First, we observe that for any collection budget, spending a fraction of it on intermediate annotations results in improved model performance, for two complex compositional datasets: DROP and Quoref. Second, these annotations encourage the model to learn the correct latent reasoning steps, helping combat some of the biases introduced during the data collection process.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “, 2018), mitigate bias (Dua et al., 2020), and even help in image and reinforcement learning settings (Andreas et al.”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, methodology): “Such intermediate annotations have been collected for DROP recently and have been shown to improve the accuracy of other models as well (Dua et al., 2020).”
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, methodology): “Prior work has used human annotations to capture coreference in question answering (Dua et al., 2020).”
Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals (2021, unknown): “p reasoning procedure [20], called multi-hopKBQA. Besides the final answer, it is also important that a multi-hop KBQA algorithm can identify a reasonable relation path leading to the answer entities [6, 27]. In some cases, even if the answer was correctly found, the relation path might be spurious.”
On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries (2020, unknown): “Dua et al. (2020) show that these annotator rationales improve model accuracy for a given annotation budget on machine reading comprehension.”
Teaching Machine Comprehension with Compositional Explanations (2020, background): “Related to our work, Dua et al. (2020) collect context spans that “should be aggregated to answer a question” and use these annotations as auxiliary supervision.”
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (2022, methodology): “Prior research has also explored using explanations as additional supervision to train joint models (Hancock et al., 2018; Dua et al., 2020; Lamm et al., 2021; Stacey et al., 2022).”
The Unreliability of Explanations in Few-Shot In-Context Learning (2022, methodology): “Prior research has also explored using explanations as additional supervision to train joint models that jointly predict the labels and explanations (Hancock et al., 2018; Dua et al., 2020; Lamm et al., 2021; Stacey et al., 2022).”