Medical Transformer: Gated Axial-Attention for Medical Image Segmentation

By Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, Vishal M. Patel. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021

1,422 citations recorded by the source corpus.

Over the past decade, Deep Convolutional Neural Networks have been widely adopted for medical image segmentation and shown to achieve adequate performance. However, due to the inherent inductive biases present in the convolutional architectures, they lack understanding of long-range dependencies in the image. Recently proposed Transformer-based architectures that leverage self-attention mechanism encode long-range dependencies and learn representations that are highly expressive. This motivates us to explore Transformer-based solutions and study the feasibility of using Transformer-based network architectures for medical image segmentation tasks. Majority of existing Transformer-based network architectures proposed for vision applications require large-scale datasets to train properly. However, compared to the datasets for vision applications, for medical imaging the number of data samples is relatively low, making it difficult to efficiently train transformers for medical applications. To this end, we propose a Gated Axial-Attention model which extends the existing architectures by introducing an additional control mechanism in the self-attention module. Furthermore, to train the model effectively on medical images, we propose a Local-Global training strategy (LoGo) which further improves the performance. Specifically, we operate on the whole image and patches to learn global and local features, respectively. The proposed Medical Transformer (MedT) is evaluated on three different medical image segmentation datasets and it is shown that it achieves better performance than the convolutional and other related transformer-based architectures. Code: https://github.com/jeya-maria-jose/Medical-Transformer

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/367f7f64ded5d18528c1013db9dfa01b075db484

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

UNETR: Transformers for 3D Medical Image Segmentation (2021, methodology): “Recently, multiple methods were proposed that explore the possibility of using transformer-based models for the task of 2D image segmentation [52, 7, 41, 51].”
Transformers in Medical Imaging: A Survey (2022, background): “[128] propose a modified gated axial attention layer [159] that works well on small medical image segmentation datasets.”
CaraNet: context axial reverse attention network for segmentation of small medical objects (2021, methodology): “Those attention methods also have been successfully used for the medical image segmentation; for example, the Medical Transformer [22] (MedT) used gated axial self-attention layer to build a Local-Global (LoGo) network for ultrasound and microscopy image segmentation, TransUNet [23] stacked selfatte”
Channel prior convolutional attention for medical image segmentation (2023, methodology): “The Gated Axial-Attention model, introduced by MedT (Valanarasu et al., 2021), extends existing architectures through the incorporation of supplementary control mechanisms within the self-attention module.”
Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, background): “Recently, transformer-based networks [24, 47, 49, 55] are proposed for medical image segmentation.”
Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video in”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “In addition, long-range dependencies are crucial for medical images because the background in medical images is generally scattered [54], thereby learning longrange dependencies between the pixels corresponding to the background can help the network in preventing misclassification [55].”
Medical Image Segmentation Review: The Success of U-Net (2022, background): “Therefore MedT [43] proposes a gated axial-attention mechanism to control the information ﬂow by positional embeddings to query, key, and value [117] in a multi-axis attention operation [118].”
DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation (2021, methodology): “Furthermore, to train the model effectively on medical images, MedT [19] introduces Gated Axial-Attention based on Axial-DeepLab [20].”
UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation (2022, background): “MedT [31] introduces a gated position-sensitive axial-attention mechanism in self-attention to control the positional embedding information in the encoder, while the ConvNet module in the decoder produces a segmentation model.”
TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions (2021, methodology): “Transformers in low-level vision: Since the introduction of Vision Transformer (ViT) [9] for visual recognition, transformers have been widely adopted for various computer vision tasks [12, 31, 49, 60, 75].”
Medical Image Segmentation Using Deep Learning: A Survey (2020, background): “Recently, a large number of researches [180] [181] [182] [183] have applied the transformer to medical image segmentation.”
nnFormer: Interleaved Transformer for Volumetric Segmentation (2021, background): “[33] proposed a gated axial-attention model (i.”