A Clockwork RNN

By Jan Koutník, Klaus Greff, Faustino J. Gomez, J. Urgen Schmidhuber. 2014

548 citations recorded by the source corpus.

Sequence prediction and classification are ubiquitous and challenging problems in machine learning that can require identifying complex dependencies between temporally distant inputs. Recurrent Neural Networks (RNNs) have the ability, in theory, to cope with these temporal dependencies by virtue of the short-term memory implemented by their recurrent (feedback) connections. However, in practice they are difficult to train successfully when long-term memory is required. This paper introduces a simple, yet powerful modification to the simple RNN (SRN) architecture, the Clockwork RNN (CW-RNN), in which the hidden layer is partitioned into separate modules, each processing inputs at its own temporal granularity, making computations only at its prescribed clock rate. Rather than making the standard RNN models more complex, CW-RNN reduces the number of SRN parameters, improves the performance significantly in the tasks tested, and speeds up the network evaluation. The network is demonstrated in preliminary experiments involving three tasks: audio signal generation, TIMIT spoken word classification, where it outperforms both SRN and LSTM networks, and online handwriting recognition, where it outperforms SRNs.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

Long Short-Term Memory-Networks for Machine Reading (2016, methodology): “The gated-feedback LSTM is a generalization of the clock-work RNN (Koutnı́k et al., 2014), with feedback gates connecting the hidden states across multiple time steps as an adaptive control of the information flow.”
Generating Long Sequences with Sparse Transformers (2019, methodology): “And for audio, in addition to (Van Den Oord et al., 2016), (Mehri et al., 2016) used a hierarchical structure and RNNs of varying clock-rates to use long contexts during inference, similar to (Koutnik et al., 2014).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…better initialization (Le et al., 2015), additional loss signal (Trinh et al., 2018), augmented memory structure (Ke et al., 2018) and others that modify the internal architecture of RNNs to ease the optimization Mikolov et al. (2014); Koutnik et al. (2014); Wu et al. (2016); Li et al. (2018).”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “Other means of enabling neural networks to learn long-range dependencies in data that have been investigated include alternative loss functions [42] and model architectures [1, 8, 14, 19, 28, 29, 32].”
Recurrent Neural Network Regularization (2014, background): “There have been a number of architectural variants of the RNN that perform better on problems with long term dependencies (Hochreiter & Schmidhuber, 1997; Graves et al., 2009; Cho et al., 2014; Jaeger et al., 2007; Koutn´ık et al., 2014; Sundermeyer et al., 2012).”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “Other neural network architectures have been described in literature with similar motivations, including the hierarchical RNN [17], clockwork RNN [18] and CNN [19].”
End-to-end attention-based large vocabulary speech recognition (2015, methodology): “In the spirit of he Clockwork RNN (Koutnik et al., 2014) and hierarchical gating RNN (Chung et al., 2015), we introduce a recurrent architecture that successively reduces source sequence length by pooling frames neighboring in time.”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “The most popular approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014; Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016).”
Gated Feedback Recurrent Neural Networks (2015, methodology): “The clockwork RNN (CW-RNN) (Koutnı́k et al., 2014) implemented this by allowing the ith module to operate at the rate of 2i−1, meaning that the module is updated only when t mod 2i−1 = 0.”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, methodology): “Like zoneout, clockwork RNNs (Koutnik et al., 2014) and hierarchical RNNs (Hihi & Bengio, 1996) update only some units’ activations at every timestep, but their updates are periodic, whereas zoneout’s are stochastic.”
Deep learning in neural networks: An overview (2014, methodology): “Gradient-based LSTM is no panacea though—other methods sometimes outperformed it at least on certain tasks (Jaeger, 2004; Schmidhuber et al., 2007; Martens and Sutskever, 2011; Pascanu et al., 2013b; Koutnı́k et al., 2014) (compare Sec.”
Recurrent Independent Mechanisms (2019, background): “Computation on demand: There are various architectures (El Hihi & Bengio, 1996; Koutnik et al., 2014; Chung et al., 2016; Neil et al., 2016; Jernite et al., 2016; Krueger et al., 2016) where parts of the LSTM’s hidden state are kept dormant at times.”
Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (2016, background): “Single-layer RNNs are trained on this data, each repeated with ﬁve random initial seeds.”
Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting (2019, background): “Many recent improvements in time series models have resulted from the alignment of architectures with unique data characteristics [13, 14].”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “…techniques for recurrent networks have been introduced and continue to be actively explored (El Hihi & Bengio, 1995; Schuster & Paliwal, 1997; Gers et al., 2002; Koutnik et al., 2014; Le et al., 2015; Ba et al., 2016; Wu et al., 2016; Krueger et al., 2017; Merity et al., 2017; Campos et al., 2018).”
Dilated Recurrent Neural Networks (2017, background): “Recent attempts have focused on multi-timescale designs, which include clockwork RNNs [12], phased LSTM [17], hierarchical multi-scale RNNs [5], etc.”
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model (2016, background): “In contrast to WaveNets, we mitigate the problem of long-term dependency with hierarchical structure and using stateful RNNs, i.e. we will always propagate hidden states to the next training sequence although the gradient of the loss will not take into account the samples in previous training sequen”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, methodology): “This idea is similar to clockwork RNNs (Koutnı́k et al., 2014), however there the top level “ticks” at a fixed, slow pace, whereas the dLSTM observes all the available training data instead.”