Focused Hierarchical RNNs for Conditional Sequence Processing

By Nan Rosemary Ke, Konrad Zolna, Alessandro Sordoni, Zhouhan Lin, Adam Trischler, Yoshua Bengio, Joelle Pineau, Laurent Charlin, C. Pal. 2018

25 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) with attention mechanisms have obtained state-of-the-art results for many sequence processing tasks. Most of these models use a simple form of encoder with attention that looks over the entire sequence and assigns a weight to each token independently. We present a mechanism for focusing RNN encoders for sequence modelling tasks which allows them to attend to key parts of the input as needed. We formulate this using a multi-layer conditional sequence encoder that reads in one token at a time and makes a discrete decision on whether the token is relevant to the context or question being asked. The discrete gating mechanism takes in the context embedding and the current hidden state as inputs and controls information flow into the layer above. We train it using policy gradient methods. We evaluate this method on several types of tasks with different attributes. First, we evaluate the method on synthetic tasks which allow us to evaluate the model for its generalization ability and probe the behavior of the gates in more controlled settings. We then evaluate this approach on large scale Question Answering tasks including the challenging MS MARCO and SearchQA tasks. Our models shows consistent improvements for both tasks over prior work and our baselines. It has also shown to generalize significantly better on synthetic tasks as compared to the baselines.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “sentence-wordcharacter), researchers have developed hierarchical RNNs to encode the temporal dependencies with different timescales using a dynamic update mechanism [172], [173].”
Exploring Interpretable LSTM Neural Networks over Multi-Variable Data (2019, methodology): “…al., 2014), have been proven to be powerful sequence modeling tools in various tasks e.g. language modelling, machine translation, health informatics, time series, and speech (Ke et al., 2018; Lin et al., 2017; Guo et al., 2016; Lipton et al., 2015; Sutskever et al., 2014; Bahdanau et al., 2014).”
Densely Connected Attention Propagation for Reading Comprehension (2018, methodology): “The competitor baselines on this dataset are Attention Sum Reader (ASR) [Kadlec et al., 2016], Focused Hierarchical RNNs (FH-RNN) [Ke et al., 2018], AMANDA [Kundu and Ng, 2018], BiDAF, AQA [Buck et al., 2017] and the Reinforced Ranker-Reader (R3) [Wang et al., 2017a].”
Revisiting Character-Based Neural Machine Translation with Capacity and Compression (2018, background): “To discourage this, we added a compression loss component similar to that of Ke et al. (2018) to penalize z activation rates outside a specified range α1, α2: Lc = ∑ l max(0, Z l − α1T, α2T − Z l), where T is source sequence length and Z l = ∑T t=1 z l t.”
Mixture Content Selection for Diverse Sequence Generation (2019, background): “Most recently, Ke et al. (2018 2018) use a pointer network (Vinyals et al., 2015) for extracting key phrases for question generation and Gehrmann et al. (2018) use content selector to limit copying probability for abstractive summarization.”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “;) or (.!?) PoWER (Goyal et al., 2020) attention drop tokens TR-BERT (Ye et al., 2021) hidden states forward tokens LAT (Kim and Cho, 2021) attention forward tokens LTP (Kim et al., 2022) attention drop tokens Transkimmer (Guan et al., 2022) hidden states forward tokens VCRNN (Jernite et al., 2017) ”