Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

By Michael M. Bronstein, Joan Bruna, Taco Cohen, Petar Velivckovi'c. 2021

1,666 citations recorded by the source corpus.

The last decade has witnessed an experimental revolution in data science and machine learning, epitomised by deep learning methods. Indeed, many high-dimensional learning tasks previously thought to be beyond reach -- such as computer vision, playing Go, or protein folding -- are in fact feasible with appropriate computational scale. Remarkably, the essence of deep learning is built from two simple algorithmic principles: first, the notion of representation or feature learning, whereby adapted, often hierarchical, features capture the appropriate notion of regularity for each task, and second, learning by local gradient-descent type methods, typically implemented as backpropagation. While learning generic functions in high dimensions is a cursed estimation problem, most tasks of interest are not generic, and come with essential pre-defined regularities arising from the underlying low-dimensionality and structure of the physical world. This text is concerned with exposing these regularities through unified geometric principles that can be applied throughout a wide spectrum of applications. Such a 'geometric unification' endeavour, in the spirit of Felix Klein's Erlangen Program, serves a dual purpose: on one hand, it provides a common mathematical framework to study the most successful neural network architectures, such as CNNs, RNNs, GNNs, and Transformers. On the other hand, it gives a constructive procedure to incorporate prior physical knowledge into neural architectures and provide principled way to build future architectures yet to be invented.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

Citations in the corpus

Geometric deep learning on molecular representations (2021, methodology): “Grids are defined by a Euclidean geometry and can be considered as a graph with a special adjacency, where (i) the vertices have a fixed ordering that is defined by the spatial dimensions of the grid, and (ii) each vertex has an identical number of adjacent edges and is therefore indistinguishable f”
VLP: A Survey on Vision-language Pre-training (2022, background): “The inspirations from mathematics include the framework of non-Euclidean space Manifold and how to put some geometric priors into the model [153, 154], which are relatively new research directions.”
The design space of E(3)-equivariant atom-centred interatomic potentials (2022, neutral): “Illustration of the construction of high body-order ACE features.”
Structure-based drug design with equivariant diffusion models (2022, positive): “Geometric Deep Learning for Drug Discovery Geometric deep learning refers to incorporating geometric priors in neural architecture design that respects symmetry and invariance, thus reduces sample complexity and eliminates the need for data augmentation (Bronstein et al., 2021).”
How Attentive are Graph Attention Networks? (2021, background): “While GAT is one of the most popular GNN architectures (Bronstein et al., 2021) and is considered as the state-of-the-art neural architecture for learning with graphs (Wang et al.”
Artificial Intelligence For Natural Product Drug Discovery (2023, background): “Deep learning has the flexibility to capture nonlinear relationships and to accept non-tabular input that extends the applicability of AI for natural product computational research to non-Euclidean domains 100,101 .”
Long Range Graph Benchmark (2022, background): “This has led to a rapid rise of interest in the development of graph neural networks (GNNs) [24, 42, 9, 61] for deep learning on geometric and graph domains.”
A Survey on Oversmoothing in Graph Neural Networks (2023, neutral): “…F ( X , G )) i = F ( X i , X j ∈N i ) operating on the multiset of 1-neighbors of each node. graph Examples of such functions used in the machine learning literature (Bronstein et al., 2021) include graph convolutions (Kipf & Welling, 2017) and attentional message passing (Velickovic et al., 2018).”
MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields (2022, background): “MPNNs [22, 9] are a type of graph neural network (GNN, [40, 4, 27, 51]) that parametrises a mapping from a labeled graph to a target space, either a graph or a vector space.”
Understanding over-squashing and bottlenecks on graphs via curvature (2021, methodology): “Multiple popular GNN architectures such as GCN (Kipf & Welling, 2017) and GAT (Veličković et al., 2018) can be posed as particular flavors of this scheme and considered instances of a more general framework of geometric deep learning (Bronstein et al., 2021).”
Challenges and opportunities in quantum machine learning (2022, background): “[92] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković, “Geometric deep learning: Grids, groups, graphs, geodesics, and gauges,” arXiv preprint arXiv:2104.”
Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere (2023, background): “An exhaustive review of these is beyond the scope of this paper and we refer the reader to the excellent overview on geometric deep learning by Bronstein et al. (2021).”
Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction (2021, methodology): “Further, contrastive learning is an elegant method to leverage inductive biases of symmetries and inductive biases [Bronstein et al., 2021].”