Discovering Symbolic Models from Deep Learning with Inductive Biases

By Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter W. Battaglia, Rui Xu, Kyle Cranmer, David N. Spergel, Shirley Ho. Neural Information Processing Systems, 2020

676 citations recorded by the source corpus.

We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example-a detailed dark matter simulation-and discover a new analytic formula which can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions extracted from the GNN using our technique also generalized to out-of-distribution data better than the GNN itself. Our approach offers alternative directions for interpreting neural networks and discovering novel physical principles from the representations they learn.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/643ac3ef063c77eb02a3d52637c11fe028bfae28

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

Neurosymbolic Program Synthesis (neutral): “For example, Cranmer et al. [15] proposed a method for symbolic regression — the automatic discovery of symbolic equations from data — and apply it to a task in cosmology (dark matter prediction).”
Graph neural networks in particle physics (2020, background): “Inspired by the success deep learning has achieved at reaching super-human performance at various tasks, various domains in the physical sciences [2], including particle physics [1, 3–6], have begun exploring deep learning as a unique tool for handling difficult scientific problems that go beyond st”
On scientific understanding with artificial intelligence (2022, methodology): “[97] Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho, “Discovering symbolic”
Lagrangian Neural Networks (2020, background): “(2019), or more specifically, the Flattened Hamiltonian Graph Network of Cranmer et al. (2020). For simplicity, we focus on 1D grids with locally-dependent dynamics (i.”
Learning Decision Trees Recurrently Through Communication (2021, background): “Beyond explainable ML, a sparse representation is considered to be essential for moving towards hybrid deep learningsymbolic models [11, 44] and for obtaining representations that are closer to conscious reasoning [2].”
Enhancing computational fluid dynamics with machine learning (2021, background): “Recent studies [54] have discussed various approaches to include interpretability in the development of deep-learning models, and one promising approach is the one proposed by Cranmer et al. [55].”
Neurosymbolic Programming (2021, methodology): “Cranmer et al. (2020) offer an example of distillation-based program learning outside the RL setting.”
End-to-end symbolic regression with transformers (2022, neutral): “It has already proved extremely useful in a variety of tasks such as inferring physical laws [1, 2].”
Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism (2022, background): “more crucial for scientists to discover the patterns from the data that induce certain predictions (Cranmer et al., 2020).”
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “To enable neuro-symbolic task planning and graph-based motion generation, we integrate recent advances in learning-based neuro-symbolic reasoning [4, 38] and object-centric dynamics model learning [18, 40].”
Utilizing graph machine learning within drug discovery and development (2021, methodology): “These methods have already been successfully applied to a variety of tasks and domains such as recommendation in social media and E-commerce [35–38], traffic estimations in Google Maps [39], misinformation detection in social media [40], and various domains of natural sciences including modelling of”
Machine Learning for Neural Decoding (2017, background): “Some recent efforts have started to investigate how ML models might be interpreted once fit to data (Ribeiro et al., 2016; Olah et al., 2018; Cranmer et al., 2020).”
Interpretable scientific discovery with symbolic regression: a review (2022, neutral): “GNN [52] Discovering Symbolic Models from DL with Inductive Biases (https://github.”