Hierarchical Attention Networks for Document Classification

By Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, Eduard H. Hovy. 2016

4,871 citations recorded by the source corpus.

We propose a hierarchical attention network for document classification. Our model has two distinctive characteristics: (i) it has a hierarchical structure that mirrors the hierarchical structure of documents; (ii) it has two levels of attention mechanisms applied at the wordand sentence-level, enabling it to attend differentially to more and less important content when constructing the document representation. Experiments conducted on six large scale text classification tasks demonstrate that the proposed architecture outperform previous methods by a substantial margin. Visualization of the attention layers illustrates that the model selects qualitatively informative words and sentences.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, background): “, 2016) 3 3 3 7* 7 WikiQA (Yang et al., 2016) 7 7 7 3 7 TREC (Voorhees and Tice, 2000) 7 3 3 3 3”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “…approaches to natural language understanding tasks like translation (Sutskever et al., 2014; Cho et al., 2014), summarization (Rush et al., 2015), and classiﬁcation (Yang et al., 2016) depend crucially on subsystems called sentence encoders that construct distributed representations for sentences.”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “To do this, Tang, Qin, and Liu (2015) ﬁrst apply a CNN or LSTM followed by a gated RNN while Yang et al. (2016) learn the sentence and document features in a hierarchical manner using a self-attention mechanism.”
Deep Pyramid Convolutional Neural Networks for Text Categorization (2017, background): “Examples are deep character-level CNNs (Zhang et al., 2015; Conneau et al., 2016), a complex combination of CNNs and recurrent neural networks (RNNs) (Tang et al., 2015), and RNNs in a word-sentence hierarchy (Yang et al., 2016).”
Rethinking Attention with Performers (2020, background): “This opens a large volume of new potential applications including: hierarchical attention networks (HANS) (Yang et al., 2016), graph attention networks (Velickovic et al., 2018), image processing (Fu et al., 2019), and reinforcement learning/robotics (Tang et al., 2020).”
The Risk of Racial Bias in Hate Speech Detection (2019, methodology): “Using a basic neural attention architecture (Wang et al., 2016; Yang et al., 2016), we train a classiﬁer initialized with GloVe vectors (Penning-ton et al., 2014) to minimize the cross-entropy of the annotated class conditional on text, x : with h = f ( x ) , where f is a BiLSTM with attention,…”
Denoising Distantly Supervised Open-Domain Question Answering (2018, background): “Yang et al. (2016) propose a hierarchical attention network which has two levels of attentions applied at the word and sentence level for document classiﬁcation.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “PARADE’s hierarchical approach follows a line of prior work on hierarchically modeling natural language text, which, to our knowledge, began in the context of deep learning with Hierarchical Attention Networks (HANs) for document classification [Yang et al., 2016].”
Stacked Cross Attention for Image-Text Matching (2018, background): “Similarly, attention methods for natural language processing adaptively select and aggregate informative snippets to infer results [42, 35, 3, 25, 28].”
MAttNet: Modular Attention Network for Referring Expression Comprehension (2018, methodology): “Given H = {ht}t=1, we apply three trainable vectors fm where m ∈ {subj, loc, rel}, computing the attention on each word [28] for each module: am,t = exp (f mht) ∑T k=1 exp (f T mhk) .”
Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm (2017, methodology): “We use a simple approach inspired by (Bahdanau et al., 2014; Yang et al., 2016) with a single parameter pr. input channel: Here h t is the representation of the word at time step t and w a is the weight matrix for the attention layer.”
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting (2019, methodology): “NN representation, we started with the fastText word embeddings as input and then trained a DNN to predict occupations in an end-to-end fashion. We used an architecture similar to that of Yang et al. [41], but with just one bi-directional recurrent neural network at the level of words and with gated”
Graph Convolutional Networks for Text Classification (2018, methodology): “To further increase the representation ﬂexibility of such models, attention mechanisms have been introduced as an integral part of models employed for text classiﬁcation (Yang et al. 2016; Wang et al. 2016).”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “…cosine similarity between the two representations. superior ability in preserving the semantic relations among different words, and thus have been successfully transferred to different tasks, such as document classiﬁcation [99], image classiﬁcation [88], and image/video caption generation [100].”
Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, background): “image classification and captioning [12, 13, 14, 15, 16, 17, 18, 19, 20], text classification [21, 22], generative models [6, 23, 24], object tracking [25], and reinforcement learning [26, 27, 28].”
Causal Interpretability for Machine Learning - Problems, Methods and Evaluation (2020, methodology): “Traditional Interpretability Interpretable Models: [106], [103], [64], [102], [65], [105], [50], [13], [40] Causal Interpretability Model-based: [77], [38], [12], [8], [112], [81], [7], Example-based: [35], [45], [39], [101], [76], [86], [73], [61], [60], [33], [34] Post-hoc: [47], [89], [66], [93],”
Interpretable Representation Learning for Healthcare via Capturing Disease Progression through Time (2018, background): “, a visit can be represented as a set of medical codes, or a patient can be represented as a set of visits) as a weighted sum of representations of its elementary objects [1, 30, 34].”