Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation

By Jianxu Chen, Lin F. Yang, Yizhe Zhang, Mark S. Alber, Danny Ziyi Chen. 2016

320 citations recorded by the source corpus.

Segmentation of 3D images is a fundamental problem in biomedical image analysis. Deep learning (DL) approaches have achieved state-of-the-art segmentation perfor- mance. To exploit the 3D contexts using neural networks, known DL segmentation methods, including 3D convolution, 2D convolution on planes orthogonal to 2D image slices, and LSTM in multiple directions, all suffer incompatibility with the highly anisotropic dimensions in common 3D biomedical images. In this paper, we propose a new DL framework for 3D image segmentation, based on a com- bination of a fully convolutional network (FCN) and a recurrent neural network (RNN), which are responsible for exploiting the intra-slice and inter-slice contexts, respectively. To our best knowledge, this is the first DL framework for 3D image segmentation that explicitly leverages 3D image anisotropism. Evaluating using a dataset from the ISBI Neuronal Structure Segmentation Challenge and in-house image stacks for 3D fungus segmentation, our approach achieves promising results comparing to the known DL-based 3D segmentation approaches.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow (2018, methodology): “While others have studied combinations of RNNs and CNNs [8, 20], we propose a hybrid architecture to decouple the prediction task from the latent space dimensions.”
A survey on deep learning in medical image analysis (2017, methodology): “Yang et al. (2016d) tackled the problem of prior / current registration in brain MRI using the OASIS data set.”
3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes (2017, methodology): “The 2D slices can also be viewed sequentially by combining a fully convolutional network (FCN) architecture with Convolutional LSTM to view the adjacent image slices as a time series to distil the 3D context from a sequence of abstracted 2D context [4].”
Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges (2019, methodology): “Chen et al. in [12] added a bidirectional CLSTM (BDC-LSTM) to a modified U-Net structure CNN. BDC-LSTM is able to capture the sequential data in two directions of z + and z − rather than single direction.”
Evaluate the Malignancy of Pulmonary Nodules Using the 3-D Deep Leaky Noisy-OR Network (2017, methodology): “Similar strategies have been used for 3-D image segmentation [16].”
A Fixed-Point Model for Pancreas Segmentation in Abdominal CT Scans (2016, methodology): “tri-planar structure (three orthogonal planes intersecting at the point to be classied) [16] or interpreting the 3D volume as a sequence of 2D image slices and applying recurrent neural networks [22][3]. A detailed discussion about 2D and 3D models can be found in [6][12]. In this paper, we choose f”
SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network (2018, background): “Moreover, it is worth emphasizing that comparing with separately trained CNN-LSTM architectures [29], [45], [46], our SV-RCNet with utilizing joint learning mechanism enables networks to take full advantage of complementary of visual and temporal information and therefore produces more discriminativ”
Deep Learning Applications in Medical Image Analysis (2018, background): “[29] combined CNN and RNN to segment neuronal and fungal structures from three-dimensional electron microscope images.”
UNETR: Transformers for 3D Medical Image Segmentation (2021, methodology): “The intuition of employing varying sizes was followed by multi-scan, multi-path models [24, 25, 8] to capture downsampled features of the image.”
Understanding Deep Learning Techniques for Image Segmentation (2019, methodology): “The approach has been implemented with object segmentation frameworks like FCN and U-NET [28].”