Induction of Multiscale Temporal Structure

By Michael C. Mozer. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “…relevant events, sev- eral sequence processing methods were proposed, including Focused BP based on decay factors for activations of units in RNNs (Mozer, 1989, 1992), Time-Delay Neural Networks (TDNNs) (Lang, Waibel, & Hinton, 1990) and their adaptive extension (Bodenhausen & Waibel, 1991),…”
Variable Computation in Recurrent Neural Networks (2016, neutral): “On the other hand, the model proposed in (Mozer, 1993) has slow-moving units as well as regular ones, where the slowness is defined by a parameter τ ∈ [0, 1] deciding how fast the representation changes by taking a convex combination of the previous and predicted hidden state.”
TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (2024, background): “Time series of different scales naturally exhibit distinct properties, where fine scales mainly depict detailed patterns and coarse scales highlight macroscopic variations (Mozer, 1991).”
Generalization in Interactive Networks: The Benefits of Inhibitory Competition and Hebbian Learning (2001, background): “…for a (sparse) distributed representation across the subset of k units. kWTA mechanisms have been analyzed for factors such as stability and convergence onto k units, and can be implemented with biologically-plausible lateral inhibition mechanisms (Majani et al., 1989; Fukai & Tanaka, 1997).”
Learning Long-Term Dependencies with (2007, unknown): “Similar results were obtained by Mozer [19].”
Switching between internal and external modes: A multiscale learning principle (2017, background): “…that, when learning multiscale structure that is nested in time, a learning system endowed with a hierarchy of slowness can be more powerful (Mozer, 1992) and more efﬁcient (Chung, Ahn, & Bengio, 2016; Schmidhuber, 1992), with higher-order structure being learned by the more slowly varying…”
Learning long-term dependencies with gradient descent is difficult (1994, unknown): “Similar results were obtained by Mozer [19].”
Hierarchical Multiscale Recurrent Neural Networks (2016, background): “…both hierarchical and temporal representation has been among the long-standing challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many temporal data (Schmidhuber, 1991; Mozer, 1993; El Hihi & Bengio, 1995; Lin et al., 1996; Koutník et al., 2014).”
A Clockwork RNN (2014, background): “One of the earliest attempts to enable RNNs to handle long-term dependencies is the Reduced Description Network (Mozer, 1992; 1994).”
Learning Unambiguous Reduced Sequence Descriptions (1991, background): “Since only unpredictable inputs convey new information, ignore all predictable inputs but let all unexpected inputs (plus information about the time step at which they occurred) become inputs to a higher-level network of the same kind (working on a slower, self-adjusting time scale).”
Understanding LSTM - a tutorial into Long Short-Term Memory Recurrent Neural Networks (2019, unknown): “RNNs are limited to look back in time for approximately ten timesteps [38], [56].”
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies (2001, methodology): “sequences causing the time lags (see also [18]).”
Supervised Sequence Labelling with Recurrent Neural Networks (2012, background): “, 1996; Plate, 1993) or time constants (Mozer, 1992), and hierarchical sequence compression (Schmidhuber, 1992).”
Bridging Long Time Lags by Weight Guessing and \Long Short Term Memory" (1996, methodology): “Within each memory cell, there is a linear unit with a (cid:12)xed-weight self-connection (compare Mozer's time constants [19]).”
Learning Sequential Tasks by Incrementally Adding Higher Orders (1992, methodology): “The values for the standard recurrent network and for Mozer's own variation are quoted from Mozer's paper [7].”