Roto-Translation Covariant Convolutional Networks for Medical Image Analysis

By Erik J. Bekkers, Maxime W. Lafarge, M. Veta, K. Eppenhof, Josien P. W. Pluim, Remco Duits. 2018

199 citations recorded by the source corpus.

We propose a framework for rotation and translation covariant deep learning using $SE(2)$ group convolutions. The group product of the special Euclidean motion group $SE(2)$ describes how a concatenation of two roto-translations results in a net roto-translation. We encode this geometric structure into convolutional neural networks (CNNs) via $SE(2)$ group convolutional layers, which fit into the standard 2D CNN framework, and which allow to generically deal with rotated input samples without the need for data augmentation. We introduce three layers: a lifting layer which lifts a 2D (vector valued) image to an $SE(2)$-image, i.e., 3D (vector valued) data whose domain is $SE(2)$; a group convolution layer from and to an $SE(2)$-image; and a projection layer from an $SE(2)$-image to a 2D image. The lifting and group convolution layers are $SE(2)$ covariant (the output roto-translates with the input). The final projection layer, a maximum intensity projection over rotations, makes the full CNN rotation invariant. We show with three different problems in histopathology, retinal imaging, and electron microscopy that with the proposed group CNNs, state-of-the-art performance can be achieved, without the need for data augmentation by rotation and with increased performance compared to standard CNNs that do rely on augmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Group Equivariant Stand-Alone Self-Attention For Vision (2020, background): “…et al., 2016; Marcos et al., 2017; Worrall et al., 2017; Weiler et al., 2018b; Li et al., 2018; Cheng et al., 2018; Hoogeboom et al., 2018; Bekkers et al., 2018; Veeling et al., 2018; Lenssen et al., 2018; Graham et al., 2020) to spherical rotations (Cohen et al., 2018; Worrall & Brostow,…”
General E(2)-Equivariant Steerable CNNs (2019, methodology): “9 C16 [7, 9] 1.”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, neutral): “; Mallat,2015;Cohen & Welling,2016;2017; Marcos et al. ,2017 ;Weiler et al. 2018b a Worrall et al. 2017;Worrall & Brostow,2018;Winkels & Cohen,2018; Veeling et al.,2018;Thomas et al.,2018;Bekkers et al., 2018;Hoogeboom et al.,2018), and the sphere (see below). Ravanbakhsh et al.(2017) st”
3D G-CNNs for Pulmonary Nodule Detection (2018, neutral): “This is a burgeoning field of research, with G-CNNs being developed for discrete 2D rotation and reflection symmetries [15, 19, 16], continuous planar rotations [20, 21, 22], 3D rotations of spherical signals [23], and permutations of nodes in a graph [24].”
Learning to Convolve: A Generalized Weight-Tying Approach (2019, methodology): “Neither the bilinear interpolation of Bekkers et al. (2018) nor the Fourier-based interpolation of Weiler et al. (2018b) satisfy unitarity.”
Wavelet Networks: Scale Equivariant Learning From Raw Waveforms (2020, methodology): “The usage of continuous smooth bases has proven advantageous for group convolutions [12, 76, 4, 75, 74, 13, 62, 3, 23].”
B-Spline CNNs on Lie Groups (2019, methodology): “…Dieleman et al., 2016; Winkels & Cohen, 2018; Worrall & Brostow, 2018; Hoogeboom et al., 2018), regular continuous G-CNNs (Oyallon & Mallat, 2015; Bekkers et al., 2015a; 2018b; Weiler et al., 2018a; Zhou et al., 2017; Marcos et al., 2017) and steerable continuous G-CNNs (Cohen et al., 2018b;…”
Deep Scale-spaces: Equivariance Over Scale (2019, unknown): “…et al., 2018a, Kondor et al., 2018, Thomas et al., 2018] and discrete roto-translations in 2D [Cohen and Welling, 2016a, Weiler et al., 2018b, Bekkers et al., 2018, Hoogeboom et al., 2018] and 3D [Worrall et al., 2017], continuous rotations on the sphere [Esteves et al., 2018, Cohen et al.,…”
Geometric and Physical Quantities improve E(3) Equivariant Message Passing (2021, background): “…that designing CNNs to exploit additional symmetries via group convolutions has even further increased their performance (Cohen & Welling, 2016; 2017; Worrall et al., 2017; Cohen et al., 2018; Kondor & Trivedi, 2018; Weiler et al., 2018; Bekkers et al., 2018; Bekkers, 2019; Weiler & Cesa, 2019).”