DCN+: Mixed Objective and Deep Residual Coattention for Question Answering

By Caiming Xiong, Victor Zhong, Richard Socher. 2017

112 citations recorded by the source corpus.

Traditional models for question answering optimize using cross entropy loss, which encourages exact answers at the cost of penalizing nearby or overlapping answers that are sometimes equally accurate. We propose a mixed objective that combines cross entropy loss with self-critical policy learning. The objective uses rewards derived from word overlap to solve the misalignment between evaluation metric and optimization objective. In addition to the mixed objective, we improve dynamic coattention networks (DCN) with a deep residual coattention encoder that is inspired by recent work in deep self-attention and residual networks. Our proposals improve model performance across question types and input lengths, especially for long questions that requires the ability to capture long-term dependencies. On the Stanford Question Answering Dataset, our model achieves state-of-the-art results with 75.1% exact match accuracy and 83.1% F1, while the ensemble obtains 78.9% exact match accuracy and 86.0% F1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “Most of the top-ranking methods on SQuAD leaderboard are based on this attention mechanism (Wang et al., 2017; Huang et al., 2017; Xiong et al., 2017; Liu et al., 2017b,a).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Many recent QA models for question answering typically assume the answer can be copied from the context [Wang and Jiang, 2017, Seo et al., 2017, Xiong et al., 2018], but this assumption does not hold for general question answering.”
A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, background): “Large-scale reading comprehension (RC) tasks that provide full super-vision for answer spans (Rajpurkar et al., 2016) have seen signiﬁcant progress recently (Seo et al., 2017; Xiong et al., 2018; Yu et al., 2018a; De-vlin et al., 2019).”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018, methodology): “petitive architectures (Liu et al., 2018; Clark and Gardner, 2017; Wang et al., 2017; Seo et al., 2017; Pan et al., 2017; Salant and Berant, 2018; Xiong et al., 2018) on SQuAD can be viewed as similar to our implemented model.”
Compositional Questions Do Not Necessitate Multi-hop Reasoning (2019, background): “Large-scale RC datasets (Hermann et al., 2015; Rajpurkar et al., 2016; Joshi et al., 2017) have enabled rapid advances in neural QA models (Seo et al., 2017; Xiong et al., 2018; Yu et al., 2018; De-vlin et al., 2018).”
Efficient and Robust Question Answering from Minimal Context over Documents (2018, methodology): “We train DCN+ (Xiong et al., 2018), one of the state-of-the-art models on SQuAD (details in Section 3.1), on the oracle sentence.”
Multi-hop Reading Comprehension through Question Decomposition and Rescoring (2019, background): “Recently, the availability of large-scale reading comprehension-datasets (Hermann et al., 2015; Rajpurkar et al., 2016; Joshi et al., 2017) has led to the development of advanced RC models (Seo et al., 2017; Xiong et al., 2018; Yu et al., 2018; Devlin et al., 2019).”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “Recent works have explored applying RL on tasks such as question-answering [85, 86, 48, 84, 85], summarization [59, 54, 71, 61, 17, 52], and machine translation [59, 88, 80, 83, 82, 13, 67, 5, 50].”
Reinforced Mnemonic Reader for Machine Comprehension (2017, methodology): “…and Courville 2016) and large-scale MC benchmark datasets (Hermann et al. 2015; Hill et al. 2016; Rajpurkar et al. 2016; Joshi et al. 2017), end-to-end neural networks have achieved promising results on MC tasks (Wang et al. 2016; Xiong, Zhong, and Socher 2017; Seo et al. 2017; Wang et al. 2017).”
Contextualized Word Representations for Reading Comprehension (2017, background): “…and Schmid-huber, 1997; Schuster and Paliwal, 1997; Cho et al., 2014), and focus model design and capacity around question-document interaction , carrying out calculations where information from both is available (Seo et al., 2016; Xiong et al., 2017b; Huang et al., 2017; Wang et al., 2017).”
Ruminating Reader: Reasoning with Gated Multi-hop Attention (2017, methodology): “Xiong et al. (2017) propose the Dynamic Coattention Network, which uses co-dependent representations of the question and the context, and iteratively updates the start and end indices to recover from local maxima and to find the optimal answer span.”
Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification (2018, background): “For the model training, Xiong et al. (2017) argues that the boundary loss encourages exact answers at the cost of penalizing overlapping answers.”
Densely Connected Attention Propagation for Reading Comprehension (2018, methodology): “, 2016], residual connections [Xiong et al., 2017; Yu et al., 2018] or simply the usage”
HEAD-QA: A Healthcare Dataset for Complex Reasoning (2019, methodology): “, 2016, 2018), where some systems achieve near human-level performance (Hu et al., 2018; Xiong et al., 2017) and often surface-level knowledge suffices to answer.”
Entity-Relation Extraction as Multi-Turn Question Answering (2019, methodology): “Answers are text spans, extracted using the now standard machine reading comprehension (MRC) framework: predicting answer spans given context (Seo et al., 2016; Wang and Jiang, 2016; Xiong et al., 2017; Wang et al., 2016b).”
RikiNet: Reading Wikipedia Pages for Natural Question Answering (2020, methodology): “Dual-Attention Layer To strengthen the information fusion from the question to the paragraphs as well as from the paragraphs to the question, we adapt a dual-attention mechanism, which has been shown effective in other MRC models (Xiong et al., 2018; Seo et al., 2017; Xiong et al., 2017).”
Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader (2019, background): “Recently, text-based QA models along (Seo et al., 2016; Xiong et al., 2017; Yu et al., 2018) have achieved remarkable performance when dealing with a single passage that is guaranteed to include the answer.”
Do Multi-hop Readers Dream of Reasoning Chains? (2019, unknown): “We take this as one baseline as many other methods (Liu et al., 2017; Xiong et al., 2017) generally have similar model architectures.”