SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks

By Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer, Max Welling. 2020

953 citations recorded by the source corpus.

We introduce the SE(3)-Transformer, a variant of the self-attention module for 3D point clouds, which is equivariant under continuous 3D roto-translations. Equivariance is important to ensure stable and predictable performance in the presence of nuisance transformations of the data input. A positive corollary of equivariance is increased weight-tying within the model, leading to fewer trainable parameters and thus decreased sample complexity (i.e. we need less training data). The SE(3)-Transformer leverages the benefits of self-attention to operate on large point clouds with varying number of points, while guaranteeing SE(3)-equivariance for robustness. We evaluate our model on a toy $N$-body particle simulation dataset, showcasing the robustness of the predictions under rotations of the input. We further achieve competitive performance on two real-world datasets, ScanObjectNN and QM9. In all cases, our model outperforms a strong, non-equivariant attention baseline and an equivariant model without attention.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Materials Chemistry, Model Architectures, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Materials Chemistry

In Model Architectures

In 3D Vision & Pose

Citations in the corpus

E(n) Equivariant Graph Neural Networks (2021, methodology): “Similarly to (Fuchs et al., 2020), we extended the Charged Particles N-body experiment from (Kipf et al., 2018) to a 3 dimensional space.”
Accurate prediction of protein structures and interactions using a 3-track neural network (2021, positive): “In the second, the averaged 1D and 2D features are fed into a final SE(3)-equivariant layer (6), and following end-to-end training from amino acid sequence to 3D coordinates, backbone coordinates are generated directly by the network Baek et al.”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, methodology): “A way to detach weights from particular kernel positions comes by parametrizing convolutional kernels as (constrained) neural networks (Thomas et al., 2018; Finzi et al., 2020; Fuchs et al., 2020).”
EquiBind: Geometric Deep Learning for Drug Binding Structure Prediction (2022, methodology): “…Euclidean symmetries into GNNs. Injecting Euclidean 3D transformations into geometric DL models has become possible using equivariant message passing layers (Cohen & Welling, 2016; Thomas et al., 2018; Fuchs et al., 2020; Satorras et al., 2021; Brandstetter et al., 2021; Batzner et al., 2021).”
GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation (2022, background): “…is ubiquitous in machine learning for atomic systems, e.g. , the vectors of atomic dipoles or forces should rotate accordingly w.r.t. the conformation coordinates (Thomas et al., 2018; Weiler et al., 2018; Fuchs et al., 2020; Miller et al., 2020; Simm et al., 2021; Batzner et al., 2021).”
Geometric and Physical Quantities improve E(3) Equivariant Message Passing (2021, methodology): “Equivariant GNNs have been proposed before as either PointConv-type (Wu et al., 2019; Kristof et al., 2017) implementations of steerable (Thomas et al., 2018; Anderson et al., 2019; Fuchs et al., 2020) or regular group convolutions (Finzi et al., 2020).”
An end-to-end deep learning method for rotamer-free protein side-chain packing (2022, methodology): “Following the conventions of ([27, 12]), we use a superscript ℓ to denote type-ℓ features of dimension 2 ℓ + 1 .”
Vector Neurons: A General Framework for SO(3)-Equivariant Networks (2021, methodology): “Recently, two approaches have been introduced to tackle this setting: Tensor Field Networks [32] and SE(3)-Transformers [13].”
Masked Inverse Folding with Sequence Transfer for Protein Representation Learning (2023, background): “DeepSequence: 41 deep mutational scanning datasets originally compiled by Riesselman et al. [2018]. These datasets each measure the effects of thousands of mutations or combinations of mutations to a parent sequence.”
Learning local equivariant representations for large-scale atomistic dynamics (2022, methodology): “The later latent MLP consist of 1 hidden layer of dimensionality [64], also using a SiLU nonlinearity.”
Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs (2022, methodology): “aij are invariant to geometric transformation [23], and therefore, we only use type-0 vectors (scalars) of message fij denoted as f (0) ij for attention.”
GemNet: Universal Directional Graph Neural Networks for Molecules (2021, background): “They were then transferred to the context of 3D point clouds and molecules by Anderson et al. [1], Thomas et al. [56], Weiler et al. [58], and further developed by Batzner et al. [4], Finzi et al. [24], Fuchs et al. [26].”
The impact of AlphaFold2 one year on (2022, neutral): “David T. Jones and Janet M. Thornton”
Everything is Connected: Graph Neural Networks (2023, methodology): “Such an analysis was done in Tensor Field Networks (Thomas et al., 2018) for point clouds, and then extended to SE(3)-Transformers for general graphs (Fuchs et al., 2020).”