A Discrete Hard EM Approach for Weakly Supervised Question Answering

By Sewon Min, Danqi Chen, Hannaneh Hajishirzi, Luke S. Zettlemoyer. 2019

170 citations recorded by the source corpus.

Many question answering (QA) tasks only provide weak supervision for how the answer should be computed. For example, TriviaQA answers are entities that can be mentioned multiple times in supporting documents, while DROP answers can be computed by deriving many different equations from numbers in the reference text. In this paper, we show it is possible to convert such tasks into discrete latent variable learning problems with a precomputed, task-specific set of possible solutions (e.g. different mentions or equations) that contains one correct option. We then develop a hard EM learning scheme that computes gradients relative to the most likely solution at each update. Despite its simplicity, we show that this approach significantly outperforms previous methods on six QA tasks, including absolute gains of 2–10%, and achieves the state-of-the-art on five of them. Using hard updates instead of maximizing marginal likelihood is key to these results as it encourages the model to find the one correct answer, which we show through detailed qualitative analysis.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “…(Chen et al., 2017) - - - 29.8 Multi-Passage BERT (Wang et al., 2019) - - - 53.0 Path Retriever (Asai et al., 2020) 31.7 - - 56.5 Graph Retriever (Min et al., 2019b) 34.7 55.8 - - Hard EM (Min et al., 2019a) 28.8 50.9 - - ORQA (Lee et al., 2019) 31.3 45.1 - 20.2 REALM (Guu et al., 2020) 38.2 - -…”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Approaches such as DrQA (Chen et al., 2017), HardEM (Min et al., 2019a), GraphRetriever (Min et al., 2019b), and PathRetriever (Asai et al., 2019) in Table 1 are in this category.”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “For the AB format, we use ROUGE-L metric (Lin et al., 2006; Min et al., 2019; Nishida et al., 2019).”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, methodology): “The first variant is the standard “open-domain” version as used e.g. by (Lee et al., 2019; Min et al., 2019b,a; Asai et al., 2019), where 1) the model is only ever trained to output a single answer; 2) if a question has multiple answers, it is only trained to predict the first answer; 3) any…”
Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, methodology): “This is commonly used in structured generation problems such as semantic parsing, program synthesis and molecule generation (Liang et al., 2013; Berant et al., 2013; Guu et al., 2017; Min et al., 2019; Zhong et al., 2020; Yang et al., 2020).”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, neutral): “We compare our system to two paradigms for QA: Open-Book systems (ORQA and Graph Retriever), which ﬁrst retrieve a set of documents from a large corpus with the question and predict a span in the retrieved document as an answer and Closed-Book system ( RELIC and T5): model answers the question from ”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, methodology): “Specifically, improvements include stronger reading comprehension models (Wang et al., 2018b; Yang et al., 2019; Xiong et al., 2020; Min et al., 2019a) and paragraph reranking models (Wang”
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (2019, methodology): “6 For Natural Questions Open, we use the English Wikipedia dump from December 20, 2018, following Lee et al. (2019) and Min et al. (2019a).”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, background): “(1) P AR R EADER reads each passage in parallel and predicts a candidate span from the passage and an answerable score (Min et al., 2019b; Wang et al., 2019).”
AmbigQA: Answering Ambiguous Open-domain Questions (2020, methodology): “Although Min et al. (2019b) outperforms Asai et al. (2020) on NQ-OPEN , it is worse on A MBIG NQ.”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, methodology): “Min et al. (2019) propose a hard EM approach to train an extractive reader model for QA tasks.”
A Memory Efficient Baseline for Open Domain Question Answering (2020, methodology): “Wang et al. (2019) applied this approach to BERT pre-trained models, while Min et al. (2019) introduced an alternative method based on expectation-maximization.”
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021, methodology): “For TriviaQA, We use the open-domain train-test splits, which correspond to the unﬁltered-train and unﬁltered-dev reading comprehension splits (Lee et al., 2019b; Min et al., 2019; Karpukhin et al., 2020a).”
Relevance-guided Supervision for OpenQA with ColBERT (2020, background): “This approach is represented in Chen et al. (2017), Min et al. (2019a), and Asai et al. (2020), among many others.”
Differentiable Reasoning over a Virtual Knowledge Base (2020, neutral): “A similar observation was made by Min et al. (2019) in weakly supervised settings.”