Hierarchical Multiscale Recurrent Neural Networks

By Junyoung Chung, Sungjin Ahn, Yoshua Bengio. 2016

568 citations recorded by the source corpus.

Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural networks, which can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that our proposed multiscale architecture can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence modelling.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Neural Machine Translation in Linear Time (2016, methodology): “…1.47 Layer-normalized LSTM (Chung et al., 2016a) 1.46 MI-LSTM (Wu et al., 2016b) 1.44 Recurrent Memory Array Structures (Rocki, 2016) 1.40 HM-LSTM (Chung et al., 2016a) 1.40 Layer Norm HyperLSTM (Ha et al., 2016) 1.38 Large Layer Norm HyperLSTM (Ha et al., 2016) 1.34 Recurrent Highway Networks…”
Character-Level Language Modeling with Deeper Self-Attention (2018, neutral): “Others experiment with modifying how the RNN cells are interconnected [6, 4].”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “In the last few years, the field of language modeling has witnessed many significant advances, including but not limited to devising novel architectures to better encode the context (Bengio et al., 2003; Mikolov et al., 2010; Zilly et al., 2016; Krause et al., 2016; Grave et al., 2016b; Dauphin et a”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “As an exception, Chung et al. (2016) utilizes Bernoulli latent variables to model the hierarchical structure in language, where the Bernoulli sampling is replaced by a thresholding operation at test time to give perplexity estimation.”
Compressive Transformers for Long-Range Sequence Modelling (2019, neutral): “7L LSTM (Graves, 2013) 1.67 LN HyperNetworks Ha et al. (2016) 1.34 LN HM-LSTM Chung et al. (2016) 1.32 ByteNet (Kalchbrenner et al., 2016) 1.31 RHN Zilly et al. (2017) 1.27 mLSTM Krause et al. (2016) 1.24 64L Transf.”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “This precedent led many to use character-level language modeling as a benchmark to evaluate neural architectures (Kalchbrenner et al., 2016; Chung et al., 2017; Ha et al., 2017; Zilly et al., 2017; Melis et al., 2018; Al-Rfou et al., 2019).”
Categorical Reparameterization with Gumbel-Softmax (2016, methodology): “For k = 2 (Bernoulli), ST Gumbel-Softmax is similar to the slope-annealed Straight-Through estimator proposed by Chung et al. (2016), but uses a softmax instead of a hard sigmoid to determine the slope.”
Dynamic Neural Networks: A Survey (2021, background): “sentence-wordcharacter), researchers have developed hierarchical RNNs to encode the temporal dependencies with different timescales using a dynamic update mechanism [172], [173].”
Recurrent Batch Normalization (2016, methodology): “1.32 Zoneout (Krueger et al., 2016) 1.27 HM-LSTM (Chung et al., 2016) 1.24 HyperNetworks (Ha et al., 2016) 1.22 1 shows that our batch-normalized LSTM achieves state of the art on both MNIST and p MNIST.”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “Furthermore, while the recurrent dropout method is speciﬁc to LSTMs and GRUs, zoneout generalizes their intuition to any model that sequentially builds distributed representations of its input.”
Recurrent Independent Mechanisms (2019, background): “Computation on demand: There are various architectures (El Hihi & Bengio, 1996; Koutnik et al., 2014; Chung et al., 2016; Neil et al., 2016; Jernite et al., 2016; Krueger et al., 2016) where parts of the LSTM’s hidden state are kept dormant at times.”
Deep Neural Decision Trees (2018, methodology): “Optionally, we can apply the slope annealing trick (Chung et al., 2017) that progressively reduces the temperature during training so that we can get a more deterministic model in the end.”
Dilated Recurrent Neural Networks (2017, background): “Recent attempts have focused on multi-timescale designs, which include clockwork RNNs [12], phased LSTM [17], hierarchical multi-scale RNNs [5], etc.”
Learning Multimodal Graph-to-Graph Translation for Molecular Optimization (2018, methodology): “This technique has been successfully applied to training neural networks with dynamic computational graphs (Chung et al., 2016).”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017, background): “While there has been much research investigating static data, such as the aforementioned ones, there is relatively little research on learning from sequential data [8, 3, 2, 9, 7, 18, 36].”