Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds

By Nathaniel Thomas, Tess E. Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, Patrick F. Riley. 2018

1,225 citations recorded by the source corpus.

We introduce tensor field neural networks, which are locally equivariant to 3D rotations, translations, and permutations of points at every layer. 3D rotation equivariance removes the need for data augmentation to identify features in arbitrary orientations. Our network uses filters built from spherical harmonics; due to the mathematical consequences of this filter choice, each layer accepts as input (and guarantees as output) scalars, vectors, and higher-order tensors, in the geometric sense of these terms. We demonstrate the capabilities of tensor field networks with tasks in geometry, physics, and chemistry.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Model Architectures, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Model Architectures

In 3D Vision & Pose

Citations in the corpus

E(n) Equivariant Graph Neural Networks (2021, methodology): “As in Tensor Field Networks we obtained better results by using the Relu activation layer instead of the Swish.”
Directional Message Passing for Molecular Graphs (2020, background): “Following work has generalized this principle to spheres (Cohen et al., 2018), molecules (Thomas et al., 2018), volumetric data (Weiler et al., 2018), and general manifolds (Cohen et al., 2019).”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, methodology): “A way to detach weights from particular kernel positions comes by parametrizing convolutional kernels as (constrained) neural networks (Thomas et al., 2018; Finzi et al., 2020; Fuchs et al., 2020).”
Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Properties (2020, background): “Meanwhile, there has also been development on neural networks which are equivariant to group action [6], some with molecules in mind [7, 8].”
SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks (2020, background): “DeepSet[30] 3DmFV[3] Set Transformer[12] PointNet[15] SpiderCNN[29] Tensor Field +z[20] PointNet++[16] SE(3)-Transf.”
Isometric Transformation Invariant and Equivariant Graph Convolutional Networks (2020, background): “Concerning point clouds, Thomas et al. (2018); Kondor (2018) discussed the way to provide rotation equivariance.”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, background): “, 2018), and point clouds (Thomas et al., 2018; Anderson et al., 2019), the requirement of working out the representation theory for the group can be cumbersome and time-consuming, and is limited to compact groups.”
General E(2)-Equivariant Steerable CNNs (2019, methodology): “The strategy followed in [39, 33, 20, 34] is based on well known analytical solutions and does not generalize to arbitrary representations.”
Cormorant: Covariant Molecular Neural Networks (2019, methodology): “We note that similar Clebsch–Gordan nonlinearities were used in [Kondor et al., 2018], and that the Clebsch–Gordan product is also an essential part of Tensor Field Networks [Thomas et al., 2018].”
Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, background): “Further examples of representations ρ that are useful in convolutional networks may be found in (Cohen & Welling, 2017; Weiler et al., 2018a; Thomas et al., 2018; Hy et al., 2018).”
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network (2018, methodology): “The Clebsch–Gordan decomposition has recently appeared in two preprints discussing neural networks for learning physical systems [21, 22].”
DiffDock: Diffusion Steps, Twists, and Turns for Molecular Docking (2022, methodology): “We use convolutional networks based on tensor products of irreducible representations (irreps) of SO(3) [18] as architecture for both the score and confidence models.”
EquiBind: Geometric Deep Learning for Drug Binding Structure Prediction (2022, methodology): “…Euclidean symmetries into GNNs. Injecting Euclidean 3D transformations into geometric DL models has become possible using equivariant message passing layers (Cohen & Welling, 2016; Thomas et al., 2018; Fuchs et al., 2020; Satorras et al., 2021; Brandstetter et al., 2021; Batzner et al., 2021).”
Learning from Protein Structure with Geometric Vector Perceptrons (2020, methodology): “The equivariance of GVP layers with respect to 3D translations and rotations also highlights a similarity to methods that leverage irreducible representations of SO (3) to deﬁne equivariant convo-lutions on point clouds (Thomas et al., 2018; Anderson et al., 2019).”
On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups (2018, background): “, 2018)), and, even more recently, since the submission of the original version of the present paper in (Thomas et al., 2018; Kondor, 2018; Weiler et al., 2018).”
GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation (2022, methodology): “…is ubiquitous in machine learning for atomic systems, e.g. , the vectors of atomic dipoles or forces should rotate accordingly w.r.t. the conformation coordinates (Thomas et al., 2018; Weiler et al., 2018; Fuchs et al., 2020; Miller et al., 2020; Simm et al., 2021; Batzner et al., 2021).”
CubeNet: Equivariance to 3D Rotation and Translation (2018, background): “Recently there has been an explosion of interest into CNNs with predefined transformation equivariances, beyond translation [8, 50, 11, 29, 16, 42, 36, 14, 19, 31, 18, 22, 15, 55, 49, 26, 25, 48, 33, 28, 9].”
Inductive biases for deep learning of higher-level cognition (2020, background): “, 2020), invariance or equivariance to known transformations (Bruna et al., 2013; Defferrard et al., 2016; Ravanbakhsh et al., 2017; Thomas et al., 2018; Finzi et al., 2020; Satorras et al., 2021) or choices of prior distributions in a Bayesian model (Jeffreys, 1946; Berger and Bernardo,”
Torsional Diffusion for Molecular Conformer Generation (2022, methodology): “The ﬁrst two closely follow the architecture of Tensor Field Networks [Thomas et al., 2018], and produce learned feature vectors for each atom.”