Sequence Labelling in Structured Domains with Hierarchical Recurrent Neural Networks

By Santiago Fernández, Alex Graves, J. Urgen Schmidhuber. 2007

100 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “10) of LSTM RNNs (Fernandez et al., 2007; Graves and Schmidhuber, 2009) trained by Connectionist Temporal Classification (CTC) (Graves et al.”
On the Binding Problem in Artificial Neural Networks (2020, background): “speech recognition (Fernández et al., 2007; Hinton et al., 2012), and even in some inherently symbolic domains such as language modeling (Devlin et al.”
Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer (2017, methodology): “We investigate encoder architectures with multitask training using hierarchical-CTC [20] with various ’hierarchies’ of CTC losses at various depths in the encoder network.”
Self-Delimiting Neural Networks (2012, methodology): “While RNNs used to be toy problem methods in the 1990s, they have recently started to beat all other methods in challenging real world applications [63, 65, 11, 21, 22, 23].”
Towards Deep Developmental Learning (2016, background): “Furthermore, hierarchical stacking of several lstms improved the capability to learn complicated sequences [122].”
A Clockwork RNN (2014, background): “Stacking LSTMs into several layers (Fernandez et al., 2007; Graves & Schmidhuber, 2009) aims for hierarchical sequence processing.”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “Hierarchical RNN architectures have also been used to discover the segmentation structure in sequences (Fernández et al., 2007; Kong et al., 2015).”
Contextual LSTM (CLSTM) models for Large scale NLP tasks (2016, result): “Capturing this hierarchical sequential structure in a language model (LM) [30] can potentially give the model more predictive accuracy, as we have seen in previous work [12, 13, 25, 33, 48].”
Hierarchical Multitask Learning for CTC-based Speech Recognition (2018, methodology): “[13], which used a hierarchical CTC model, similar to ours although only two layers deep, with phoneme label prediction as an auxiliary task in training a spoken digit sequence recognizer.”
Recurrent Highway Networks (2016, unknown): “t set (without dynamic evaluation). A variational RHN with recurrence depth of 5 comfortably outperforms all other methods. (*) Model size based on our estimate. Model BPC Size Test Data Stacked LSTM [38] 1:67 27.0M last 4 MB GF-RNN [24] 1:58 20.0M last 5 MB Grid-LSTM [20] 1:47 16.8M last 5 MB MI-LS”
Direct Acoustics-to-Word Models for English Conversational Speech Recognition (2017, methodology): “Phone CTC models and hierarchical CTC [19].”
Understanding LSTM - a tutorial into Long Short-Term Memory Recurrent Neural Networks (2019, unknown): “The most important to note are Multidimensional LSTM ( [29], [35]), Stacked LSTM ([18], [33], [68]).”
Sequence Classification Restricted Boltzmann Machines With Gated Units (2020, methodology): “In particular, we choose Bidirectional LSTM (BiLSTM) [38], [46] and Stacked LSTM (StackedLSTM) [47], [48].”
Speech Model Pre-training for End-to-End Spoken Language Understanding (2019, methodology): “By using phonemes as intermediate pre-training targets [20, 35, 36], we are able to pre-train on speech segments with no word label.”
An Application of Recurrent Neural Networks to Discriminative Keyword Spotting (2007, methodology): “The al-gorithm [9, 10] is a general framework for labelling unsegmented sequential data with artiﬁcial neural networks.”