An Artificial Agent for Robust Image Registration

By Rui Liao, Shun Miao, Pierre de Tournemire, Sasa Grbic, Ali Kamen, Tommaso Mansi, Dorin Comaniciu. 2016

207 citations recorded by the source corpus.

3-D image registration, which involves aligning two or more images, is a critical step in a variety of medical applications from diagnosis to therapy. Image registration is commonly performed by optimizing an image matching metric as a cost function. However this task is challenging due to the non-convex nature of the matching metric over the plausible registration parameter space and insufficient approches for a robust optimization. As a result, current approaches are often customized to a specific problem and sensitive to image quality and artifacts. In this paper, we propose a completely different approach to image registration, inspired by how experts perform the task. We first cast the image registration problem as a "strategic learning" process, where the goal is to find the best sequence of motion actions (e.g. up, down, etc) that yields image alignment. Within this approach, an artificial agent is learned, modeled using deep convolutional neural networks, with 3D raw image data as the input, and the next optimal action as the output. To copy with the dimensionality of the problem, we propose a greedy supervised approach for an end-to-end training, coupled with attention-driven hierarchical strategy. The resulting registration approach inherently encodes both a data-driven matching metric and an optimal registration strategy (policy). We demonstrate on two 3-D/3-D medical image registration examples with drastically different nature of challenges, that the artificial agent outperforms several state-of-the-art registration methods by a large margin in terms of both accuracy and robustness.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Robust Non-rigid Registration Through Agent-Based Action Learning (2017, methodology): “To have a stochastic exploration of the parameter space, predicted actions a t are selected given ﬁxed probabilities (see [4]).”
Diagnose like a Radiologist: Attention Guided Convolutional Neural Network for Thorax Disease Classification (2018, background): “The noticeable progress in deep learning has beneﬁted many trials in medical image analysis, such as lesion segmentation or detection [1], [2], [3], [4], [5], diseases classiﬁcation [6], [7], [8], [9], noise induction [10], image annotation [11], [12], registration [13], regression [14] and so on.”
A Gentle Introduction to Deep Learning in Medical Image Processing (2018, background): “Liao et al. propose to do so for rigid registration predicting the next optimal movement in order to align both volumes [85].”
Real-time Deep Pose Estimation with Geodesic Loss for Image-to-Template Rigid Registration (2018, background): “[14] proposed a reinforcement learning algorithm for a CNN with 3 fully connected layers.”
Nonrigid Image Registration Using Multi-scale 3D Convolutional Neural Networks (2017, methodology): “[3] used CNN regression to model a sequence of motion actions for 3D registration.”
Evaluating Reinforcement Learning Agents for Anatomical Landmark Detection (2019, unknown): “Because of the recent success of employing Q-learning in several medical imaging applications [5, 9, 25, 26, 28, 29, 30, 31], we adopt in this paper the common strategy of Q-learning based methods as a solution for the RL problem formulation of landmark detection.”
3D/2D model-to-image registration by imitation learning for cardiac procedures (2018, background): “The distance between two transformations T1 and T2 is D(T1, T2), the L2 norm of the parameters of the transformations, as described in [9].”
Deep learning in radiology: an overview of the concepts and a survey of the state of the art with focus on MRI (2018, methodology): “In [107], the authors trained a CNN-based model to learn the sequence of movements that resulted in the misalignment of the image pairs of CT and cone-beam CT examinations of the abdominal spine and heart.”
End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network (2017, methodology): “[6] used a ConvNet for intra-patient rigid registration of CT to cone-beam CT applied to either cardiac or abdominal images.”