FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension

By Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, Weizhu Chen. 2017

187 citations recorded by the source corpus.

This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives. First, it puts forward a novel concept of "history of word" to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation. Second, it introduces an improved attention scoring function that better utilizes the "history of word" concept. Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer. We apply FusionNet to the Stanford Question Answering Dataset (SQuAD) and it achieves the first position for both single and ensemble model on the official SQuAD leaderboard at the time of writing (Oct. 4th, 2017). Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “Huang et al. introduce a fully-aware attention mechanism with a novel history-of-word concept (Huang et al., 2017).”
U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “…of deep learning techniques and large-scale benchmarks (Hermann et al. 2015; Hill et al. 2015; Rajpurkar et al. 2016), the end-to-end neural methods have achieved promising results on MRC task (Seo et al. 2016; Huang et al. 2017; Chen et al. 2017; Clark and Gardner 2017; Hu et al. 2017).”
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, methodology): “Ensemble learning is an effective approach to improve model generalization, and has been used to achieve new state-of-the-art results in a wide range of natural language understanding (NLU) tasks, including question answering and machine reading comprehension (Devlin et al., 2018; Liu et al., 2018; ”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Recent approaches to MNLI use methods developed on SNLI and have even pointed out the similarities between models for question answering and NLI [Huang et al., 2017].”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, background): “In turn, these datasets have spurred a diverse array of model architecture improvements (Seo et al., 2016; Hu et al., 2017; Wang et al., 2017; Clark and Gard-ner, 2017; Huang et al., 2018).”
The Effect of Natural Distribution Shift on Question Answering Models (2020, neutral): “FusionNet (Single) (Huang et al., 2018) https://worksheets.”
Efficient and Robust Question Answering from Minimal Context over Documents (2018, methodology): “We compare with GNR (Raiman and Miller, 2017), FusionNet (Huang et al., 2018) and FastQA (Weissenborn et al.”
Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA (2019, background): “They report that the performances of state-of-the-art QA models (Seo et al., 2017; Hu et al., 2018; Huang et al., 2018) drop significantly when evaluated on the adversarial data.”
Contextualized Word Representations for Reading Comprehension (2017, background): “, 2014), and focus model design and capacity around question-document interaction, carrying out calculations where information from both is available (Seo et al., 2016; Xiong et al., 2017b; Huang et al., 2017; Wang et al., 2017).”
A Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension (2018, background): “Test (Salant and Berant, 2017) 77.58 84.16 O - QANet (Yu et al., 2018) 76.24 84.60 O O SAN (Liu et al., 2017b) 76.83 84.40 O O FusionNet (Huang et al., 2017b) 75.97 83.90 O O RaSoR + TR (Salant and Berant, 2017) 75.79 83.26 O - Conducter-net (Liu et al., 2017a) 74.41 82.74 O O Reinforced Mnemonic Re”
Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension (2018, unknown): “However, we can still bene-ﬁt from self-attention, which has become crucial for machine translation (Vaswani et al., 2017) and QA (Huang et al., 2018; Yu et al., 2018).”
Weaver: Deep Co-Encoding of Questions and Documents for Machine Reading (2018, methodology): “…published architectures: it is better than Conductor-net (Liu et al., 2017a) and Reinforced Mnemonic Reader (M-Reader + RL) (Hu et al., 2017) and outperformed by DCN+ (Xiong et al., 2017), FusionNet (Huang et al., 2017) and SAN (Liu et al., 2017b M-Reader + RL that was also applied on TriviaQA.”
Robust Machine Comprehension Models via Adversarial Training (2018, methodology): “Since then, the FusionNet model (Huang et al., 2018) used history-of-word representations and multi-level attention mechanism to obtain an improved 51.”
FlowQA: Grasping Flow in History for Conversational Machine Comprehension (2018, methodology): “We use fully-aware attention (Huang et al., 2018), which concatenates all layers of hidden vectors and uses S(x, y) = ReLU(Ux)DReLU(Uy) to compute the attention score between x, y, where U,D are trainable parameters and D is a diagonal matrix.”