Multi-dimensional Recurrent Neural Networks

By Alex Graves, Santiago Fernández, J. Urgen Schmidhuber. 2007

323 citations recorded by the source corpus.

Recurrent neural networks (RNNs) have proved effective at one dimensional sequence learning tasks, such as speech and online handwriting recognition. Some of the properties that make RNNs suitable for such tasks, for example robustness to input warping, and the ability to access contextual information, are also desirable in multi-dimensional domains. However, there has so far been no direct way of applying RNNs to data with more than one spatio-temporal dimension. This paper introduces multi-dimensional recurrent neural networks, thereby extending the potential applicability of RNNs to vision, video processing, medical imaging and many other areas, while avoiding the scaling problems that have plagued other multi-dimensional models. Experimental results are provided for two image segmentation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Convolutional Neural Fabrics (2016, methodology): “Finally, we analyze the number of response maps, parameters, and activations of fabrics in Section 3.3.”
Transition-Based Dependency Parsing with Stack Long Short-Term Memory (2015, background): “LSTMs address this with an extra memory “cell” ( c t ) that is constructed as a linear combination of the previous state and signal from the input.”
MelNet: A Generative Model for Audio in the Frequency Domain (2019, methodology): “This approach differs from the tightly coupled multidimensional recurrences used by MDRNNs [17, 16] and Grid LSTMs [25] and more closely resembles the approach taken by ReNet [52].”
Spatio-temporal video autoencoder with differentiable memory (2015, background): “Multi-dimensional LSTM networks have been proposed to deal with (2D) images Graves et al. (2007) or (3D) volumetric data Stollenga et al. (2015), treating the data as spatial sequences.”
Learning Affinity via Spatial Propagation Networks (2017, background): “The one-way connection is a direct extension of sequential recurrent propagation [8, 31, 13].”
Semantic Object Parsing with Local-Global Long Short-Term Memory (2015, background): “Instead of pure local factorized LSTMs in [16] [10], the features of each position in the proposed LG-LSTM are inﬂuenced by the short-distance dependencies as well as the long-distance global information from the whole image.”
A review on the long short-term memory model (2020, background): “Keywordspottingisadetectiontaskconsistingindiscoveringthe presence of speciﬁc spoken words in speech signals (Fernández et al. 2007).”
Deep Learning for Audio Signal Processing (2019, background): “Frequency LSTMs (F-LSTM) [36] and Time-Frequency LSTMs (TF-LSTM) [37]–[39] have been introduced as alternatives to CNNs to model correlations in frequency.”
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation (2018, methodology): “CNN for semantic segmentation: Different CNN-based segmentation networks have been proposed, such as multi-dimensional recurrent neural networks [38], encoderdecoders [20,21,39,40], hypercolumns [41], region-based representations [42,43], and cascaded networks [44].”
Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation (2016, background): “To deal with the above problem, we propose word-lattice based GRU encoders for NMT. Similar to the dominant NMT model (Bahdanau, Cho, and Bengio 2015), our encoders are bidirectional RNNs with GRU.”
Exploiting time-frequency patterns with LSTM-RNNs for low-bitrate audio restoration (2019, background): “Rather than separately capturing the frequency and time context in a cascaded structure, several studies try to seek help from a more general RNN architecture speciﬁcally designed for multi-dimensional sequence processing, i. e., multi-dimensional RNNs (MDRNNs) [13].”
Machine Learning for Fluid Mechanics (2019, background): “An extended architecture, called the multi-dimensional LSTM network (MD-LSTM) (Graves et al. 2007), was proposed to eﬃciently handle high-dimensional spatiotemporal data.”
Recurrent Neural Aligner: An Encoder-Decoder Neural Network Model for Sequence to Sequence Mapping (2017, methodology): “There has been previous work on merging or combining recurrent neural network states for multi-dimensional RNN [15] and Grid LSTM [16].”
Grid Long Short-Term Memory (2015, background): “Grid LSTM with three or more dimensions is analogous to Multidimensional LSTM [14, 35, 13, 12], but differs from it not just by having the cells along the depth dimension, but also by using the proposed mechanism for modulating the N -way interaction that is not prone to the instability present in…”
A Review on Deep Learning Techniques for Video Prediction (2020, background): “d Gated Recurrent Unit (GRU) [87], these problems were mitigated. Shi et al. extended the use of LSTM-based models to the image space [13]. While some authors explored multidimensional LSTM (MD-LSTM) [88], others stacked recurrent layers to capture abstract spatio-temporal correlations [49], [89]. Z”
Deep Neural Network for Structural Prediction and Lane Detection in Traffic Scene (2017, methodology): “For image analysis, LSTM was extended to multiple directions and dimensions and applied to handwritten digits recognition [34].”
Supervised Sequence Labelling with Recurrent Neural Networks (2012, background): “LSTM01 LSTM原理与应用及训练方法_Constant dripping wears …Accepted papers | EMNLP 2021roberta-base · Hugging Face有哪些LSTM(Long Short Term Memory)和RNN(Recurrent)网络的 …(PDF) Deep Learning - ResearchGateFake news detection: A hybrid CNN-RNN based deep learning GitHub - shawnyuen IEEE ICIP 2021 || Anchorage, Alaska,”
A survey on deep learning techniques for image and video semantic segmentation (2018, unknown): “[19] proposed a Multi-dimensional Recurrent Neural Network (MDRNN) architecture which replaces each single recurrent connection from standard RNNs with d connections, where d is the number of spatio-temporal data dimensions.”