Learning Long-Term Dependencies with

By Patrice Simardy, Paolo FrasconizyAT. 2007

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “Compare also more recent RNN algorithms able to deal with long time lags (Schäfer et al., 2006; Martens and Sutskever, 2011; Zimmermann et al., 2012; Koutnı́k et al., 2014).”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “…al., 2017; Lin et al., 2018; Chitsaz et al., 2020; Goldberg et al., 2020), Recurrent Neural Networks (Koplon and Sontag, 1997; Zhang and Chan, 2000; Zhang et al., 2018), Trans-formers (Choromanski et al., 2020; Tamkin et al., 2020), and MLP layers more generally (Cheng et al., 2015; Moczulski et…”
Transfer learning for biomedical named entity recognition with neural networks (2018, background): “Although RNNs can, in theory, learn long-range dependencies, they fail to do so in practice and tend to be biased towards their most recent inputs in the sequence (Bengio et al., 1994).”
Reward-based training of recurrent neural networks for cognitive and value-based tasks (2016, methodology): “In order to take advantage of recent developments in training RNNs [in particular, addressing the problem of vanishing gradients (Bengio et al., 1994)] while retaining intepretability, we use a modified form of Gated Recurrent Units (GRUs) (Cho et al., 2014; Chung et al., 2014) with a…”
Crop Yield Prediction Using Deep Neural Networks (2019, background): “Increasing the number of hidden layers may reduce the classification or regression errors, but it may also cause the vanishing/exploding gradients problem that prevents the convergence of the neural networks (Bengio et al., 1994; Glorot and Bengio, 2010; He et al., 2016).”
Theoretical Limitations of Self-Attention in Neural Sequence Models (2019, background): “While theoretical work has investigated the power of recurrent neural networks in depth (e.g., Siegelman and Sontag (1995); Bengio et al. (1994); Weiss et al. (2018); Miller and Hardt (2019); Merrill (2019)), the theoretical study of self-attention has begun only recently (Pérez et al., 2019; Hsieh…”
Stochastic variational learning in recurrent spiking networks (2014, background): “Learning the synapses between hidden neurons in recurrent models has been recognized as a difﬁcult problem, as these synapses only contribute indirectly to the activity of the visible neurons and as recurrence leads to the “vanishing gradient” problem (Bengio et al., 1994).”
Joint entity recognition and relation extraction as a multi-head selection problem (2018, methodology): “In this work, we use multi-layer LSTMs, a specific kind of RNNs which are able to capture long term dependencies well (Bengio et al., 1994; Pascanu et al., 2013).”