Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation

By Marijn F. Stollenga, Wonmin Byeon, Marcus Liwicki, J. Urgen Schmidhuber. 2015

300 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) can be shifted across 2D images or 3D videos to segment them. They have a fixed input size and typically perceive only small local contexts of the pixels to be classified as foreground or background. In contrast, Multi-Dimensional Recurrent NNs (MD-RNNs) can perceive the entire spatio-temporal context of each pixel in a few sweeps through all pixels, especially when the RNN is a Long Short-Term Memory (LSTM). Despite these theoretical advantages, however, unlike CNNs, previous MD-LSTM variants were hard to parallelise on GPUs. Here we re-arrange the traditional cuboid order of computations in MD-LSTM in pyramidal fashion. The resulting PyraMiD-LSTM is easy to parallelise, especially for 3D data such as stacks of brain slice images. PyraMiD-LSTM achieved best known pixel-wise brain image segmentation results on MRBrainS13 (and competitive results on EM-ISBI12).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Pixel Recurrent Neural Networks (2016, methodology): “The first type is the Row LSTM layer where the convolution is applied along each row; a similar technique is described in (Stollenga et al., 2015).”
A survey on deep learning in medical image analysis (2017, methodology): “Stollenga et al. (2015) where the ﬁrst to use a 3D LSTM-RNN with convolutional layers in six directions.”
Spatio-temporal video autoencoder with differentiable memory (2015, methodology): “, 2007) or (3D) volumetric data (Stollenga et al., 2015), treating the data as spatial sequences.”
Deep learning based brain tumor segmentation: a survey (2020, methodology): “Similar to [95] using parallel multi-dimensional LSTMs for brain MR segmentation, each C-GRU can only handle one direction and one dimensional signals.”
A Fixed-Point Model for Pancreas Segmentation in Abdominal CT Scans (2016, methodology): “g a tri-planar structure (three orthogonal planes intersecting at the point to be classied) [16] or interpreting the 3D volume as a sequence of 2D image slices and applying recurrent neural networks [22][3]. A detailed discussion about 2D and 3D models can be found in [6][12]. In this paper, we choo”
Deep Learning Applications in Medical Image Analysis (2018, background): “[30] segmented both three-dimensional electron microscope images of neurons as well as MRI brain scans.”
Learning Recursive Filters for Low-Level Vision via a Hybrid Neural Network (2016, methodology): “To design a data-driven RNN filter, a straightforward approach is to take each pixel as a hidden recurrent node in a two-dimensional (2D) spatial sequence [15,16,17], and use the recurrent structure to learn the propagation weight matrix.”
Semantic Object Parsing with Graph LSTM (2016, background): “, 2 or 8 neighboring pixels [1][32][21][33] or diagonal neighborhood [2][19]), where diverse natural visual correlations (e.”
Deep learning in bioinformatics (2016, methodology): “Gene expression regulation [99-104] Anomaly classification [125-132] Brain decoding [164-167] Segmentation [134-140] Anomaly classification [176] Recognition [144-147]”
Neural Attribute Machines for Program Generation (2017, background): “of RNN is the Long Short Term Memory (LSTM) cell [14], which has been favored in recent literature for its ability to learn long-term dependences, although the NAM framework is general to any type of RNN cell [15, 16, 17].”
Applications of Deep Learning and Reinforcement Learning to Biological Data (2017, methodology): “2G), RNN with DA outperformed other non-DL methods (RNNAE: 0.97 vs. [MIL: 0.58, PFM: 0.77, PFM: 0.59; PBM: 0.58, EN: 0.71, and MDS: 0.64]) [100].”
A review on the long short-term memory model (2020, methodology): “In order to tackle the problem of parallelisation on GPUs of multidimensional LSTMs, Stollenga et al. (2015) proposed the PyraMiD-LSTM, a re-arrangement of the traditional cuboid order of computations in a pyramidal fashion.”
ReSeg: A Recurrent Neural Network for Object Segmentation (2015, background): “Recently however many approaches in this direction have been explored (see, e.g., Graves & Schmidhuber, 2009; Grangier et al., 2009; Farabet et al., 2013; Hariharan et al., 2014; Long et al., 2014; Ronneberger et al., 2015; Pinheiro & Collobert, 2014; Gatta et al., 2014; Visin et al., 2015; Kalchbre”
Dense Recurrent Neural Networks for Scene Labeling (2018, background): “scene labeling. The work of [6] explores the two-dimensional long short term memory (LSTM) networks for scene parsing by taking into account the complex spatial dependencies of pixels in an image. In [43], Stollenga et al. introduce a parallel multi-dimensional LSTM for image segmentation. Liang et ”
Exploiting time-frequency patterns with LSTM-RNNs for low-bitrate audio restoration (2019, background): “The underlying idea of MDRNNs is to replace the single recurrent connection found in standard RNNs with as many recurrent connections as there are dimensions in the data [13] and have efficiently applied to, for example, image segmentation [57].”