Learning What is Essential in Questions

By Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Dan Roth. 2017

37 citations recorded by the source corpus.

Question answering (QA) systems are easily distracted by irrelevant or redundant words in questions, especially when faced with long or multi-sentence questions in difficult domains. This paper introduces and studies the notion of essential question terms with the goal of improving such QA solvers. We illustrate the importance of essential question terms by showing that humans’ ability to answer questions drops significantly when essential terms are eliminated from questions.We then develop a classifier that reliably (90% mean average precision) identifies and ranks essential terms in questions. Finally, we use the classifier to demonstrate that the notion of question term essentiality allows state-of-the-art QA solver for elementary-level science questions to make better and more informed decisions,improving performance by up to 5%.We also introduce a new dataset of over 2,200 crowd-sourced essential terms annotated science questions.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Improving Question Answering with External Knowledge (2019, methodology): “Ni et al. study improving information retriever by essnetial terms ( Khashabi et al., 2017).”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “This yields the following datasets: CosmosQA (Huang et al., 2019), DROP (Dua et al., 2019), EssentialTerms (Khashabi et al., 2017), MCTACO (Zhou et al., 2019), MultiRC (Khashabi et al., 2018), QASC (Khot et al., 2020), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019) and Winogrande (Sakaguchi ”
QASC: A Dataset for Question Answering via Sentence Composition (2019, background): “Training a model on our annotations or essential terms (Khashabi et al. 2017) could help better identify these concepts.”
SciTaiL: A Textual Entailment Dataset from Science Question Answering (2018, background): “Identifying essential terms (Khashabi et al. 2017) from the hypothesis could be one way to mitigate this problem.”
Improving Machine Reading Comprehension with General Reading Strategies (2018, methodology): “, 2018) uses essential terms (Khashabi et al., 2017) and Finetuned QACNN (Chung et al.”
A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset (2018, background): “Khashabi et al. (2017) showed that an IR based QA system can be improved substantially by focusing on terms which are essential in question.”
KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning (2019, background): “How to efficiently retrieve contextually-related knowledge from noisy knowledge resources is still an open research question by itself (Weissenborn et al., 2017; Khashabi et al., 2017), and thus most prior works choose to stop here (Zhong et al.”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “After discussing general observations on CFT in Section 4.3.1, we will delve into each sub-category in detail.”
Answering Science Exam Questions Using Query Rewriting with Background Knowledge (2018, methodology): “ 0.88 0.74 0.80 0.77 TransH 0.88 0.75 0.77 0.76 PPMI 0.87 0.77 0.72 0.75 Figure 3: Essential terms classiﬁcation performance; (Acc)uracy, (Pr)ecision, (Re)call, and (F1) score for the ET Classiﬁer of Khashabi et al. (2017) reﬂect the 70/9/21 train/dev/test split reported in their paper. As ET Net do”
Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering (2018, methodology): “A similar outcome has been reported in Jansen et al. (2017); Khashabi et al. (2017) where essential term extraction methods have shown around 2%-4% gain compared with TF-IDF models and struggle to obtain further improvement on SQA tasks.”
Controlling Information Aggregation for Complex Question Answering (2018, methodology): “To mitigate this issue, recent QA methods that operate over OIE graphs limit themselves to reasoning with artiﬁcially short paths (Khot et al. 2017; Khashabi et al. 2017).”
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions (2021, methodology): “In particular, we use the templates used for building the following datasets: CosmosQA (Huang et al., 2019), DROP (Dua et al., 2019), Essential-Terms (Khashabi et al., 2017), MCTACO (Zhou et al., 2019), MultiRC (Khashabi et al., 2018), QASC (Khot et al., 2020), Quoref (Dasigi et al., 2019), and Wino”
Careful Selection of Knowledge to Solve Open Book Question Answering (2019, methodology): “Related to our work for extracting missing knowledge are the works of (Ni et al., 2018; Musa et al., 2018; Khashabi et al., 2017) which respectively generate a query either by extracting key terms from a question and an answer option or by classifying key terms or by Seq2Seq models to generate key t”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, background): “Jia & Liang (2017) have shown that neural question answering systems are largely affected by adversarial distracting sentences, whereas follow up work (Khashabi et al., 2017; Ni et al., 2019) proposes learning strategies that mitigate the problem.”
Improving Retrieval-Based Question Answering with Deep Inference Models (2018, methodology): “The neural network has been trained on the dataset collected by Khashabi et al. [14].”