Bidirectional Attention Flow for Machine Comprehension

By Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, Hannaneh Hajishirzi. 2016

2,143 citations recorded by the source corpus.

Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query. Recently, attention mechanisms have been successfully extended to MC. Typically these methods use attention to focus on a small portion of the context and summarize it with a fixed-size vector, couple attentions temporally, and/or often form a uni-directional attention. In this paper we introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bi-directional attention flow mechanism to obtain a query-aware context representation without early summarization. Our experimental evaluations show that our model achieves the state-of-the-art results in Stanford Question Answering Dataset (SQuAD) and CNN/DailyMail cloze test.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “Table 2 shows top leaderboard entries as well as results from top published systems (Seo et al., 2017; Clark and Gardner, 2018; Peters et al., 2018a; Hu et al., 2018).”
Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “propose a bi-directional attention flow network to capture the relevance at different levels of granularity (Seo et al., 2016).”
U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “…of deep learning techniques and large-scale benchmarks (Hermann et al. 2015; Hill et al. 2015; Rajpurkar et al. 2016), the end-to-end neural methods have achieved promising results on MRC task (Seo et al. 2016; Huang et al. 2017; Chen et al. 2017; Clark and Gardner 2017; Hu et al. 2017).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “2017), question answering (Seo et al., 2016; Xiong et al., 2016), and natural language inference1.”
Deep Contextualized Word Representations (2018, background): “Our baseline model (Clark and Gardner, 2017) is an improved version of the Bidirectional Attention Flow model in Seo et al. (BiDAF; 2017).”
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “Table 4: Speed comparison between our model and BiDAF (Seo et al., 2016) on SQuAD dataset.”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “We apply this approach with a model design that integrates some recent ideas in reading comprehension models, including self-attention (Cheng et al., 2016) and bi-directional attention (Seo et al., 2016).”
Learned in Translation: Contextualized Word Vectors (2017, methodology): “Recent work in NLP has continued in this direction by using pretrained word representations to improve models for entailment [Bowman et al., 2014], sentiment analysis [Socher et al., 2013], summarization [Nallapati et al., 2016], and question answering [Seo et al., 2017, Xiong et al., 2017].”
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, methodology): “Finally, we present baseline experiments on the TriviaQA dataset, including a linear classiﬁer inspired by work on CNN Dailymail and MCTest (Chen et al., 2016; Richardson et al., 2013) and a state-of-the-art neural network baseline (Seo et al., 2017).”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Following previous reading comprehension work (Rajpurkar et al., 2016; Seo et al., 2016; Lee et al., 2016; Clark & Gardner, 2017), we will assume that the answer y can be found as a contiguous sequence of tokens in some document z.”
NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “In recent years, researchers have proposed lots of MRC models (Chen et al., 2016; Dhingra et al., 2017; Cui et al., 2017; Seo et al., 2017) and these models have achieved remarkable results in various public benchmarks such as SQuAD (Rajpurkar et al.”
Technical report on Conversational Question Answering (2019, methodology): “Many MRC models have been proposed to tackle single-turn QA, such as BiDAF(Seo et al., 2016), DrQA(Chen et al., 2017), R-Net(Wang et al., 2017) and QANet(Yu et al., 2018) in SQuAD dataset.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, methodology): “…instructions given to workers, and partially through the use of an adversarial baseline : we ran a base-line reading comprehension method (BiDAF) (Seo et al., 2016) in the background as crowd workers were writing questions, requiring them to give questions that the baseline system could not…”
Learning and Evaluating General Linguistic Intelligence (2019, methodology): “We then use a bidirectional attention flow network (BiDAF; Seo et al., 2017) to aggregate context (premise) and question (hypothesis) representations to predict an answer span (a label) in question answering (natural language inference).”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, methodology): “It consists of components such as bidirectional attention flow (Seo et al., 2016) and self attention which are widely used in MRC models.”
CoQA: A Conversational Question Answering Challenge (2018, background): “The state-of-the-art reading comprehension models for extractive question answering focus on finding a span in the passage that matches the question best (Seo et al., 2016; Chen et al., 2017; Yu et al., 2018).”
QuAC: Question Answering in Context (2018, methodology): “BiDAF++ We use a re-implementation of a top-performing SQuAD model (Peters et al., 2018) that augments bidirectional attention ﬂow (Seo et al., 2016, BiDAF) with self-attention (Clark and Gardner, 2018) and contextualized embeddings.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “Many recent QA models for question answering typically assume the answer can be copied from the context [Wang and Jiang, 2017, Seo et al., 2017, Xiong et al., 2018], but this assumption does not hold for general question answering.”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, background): “In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gardner, 2017; Huang et al., 2018).”