Extrapolation and learning equations

By Georg Martius, Christoph H. Lampert. 2016

213 citations recorded by the source corpus.

In classical machine learning, regression is treated as a black box process of identifying a suitable function from a hypothesis set without attempting to gain insight into the mechanism connecting inputs and outputs. In the natural sciences, however, finding an interpretable function for a phenomenon is the prime goal as it allows to understand and generalize results. This paper proposes a novel type of function learning network, called equation learner (EQL), that can learn analytical expressions and is able to extrapolate to unseen domains. It is implemented as an end-to-end differentiable feed-forward network and allows for efficient gradient based training. Due to sparsity regularization concise interpretable expressions can be obtained. Often the true underlying source expression is identified.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A theory of independent mechanisms for extrapolation in generative models (2020, background): “extrapolations explored in the context of dynamical systems (Martius and Lampert 2016).”
KAN: Kolmogorov-Arnold Networks (2024, unknown): “There are many off-the-shelf symbolic regression methods based on genetic algorithms (Eureka [73], GPLearn [74], PySR [75]), neural-network based methods (EQL [76], OccamNet [77]), physics-inspired method (AI Feynman [20, 21]), and reinforcement learning-based methods [78].”
PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network (2018, methodology): “Our design of SymNetm is motivated by EQL/EQL ÷ proposed by [41, 42].”
Learning Equations for Extrapolation and Control (2018, methodology): “Figure adapted from Martius & Lampert (2016).”
Interpretable scientific discovery with symbolic regression: a review (2022, unknown): “In the case of a pure polynomial basis, the correct terms of the Taylor expansion of both cos( x 1 ) and sin( x 2 ) are identiﬁed with only approximate values of their coeﬃcients, i.e., ˆ y 1 = (0 .”
End-to-end symbolic regression with transformers (2022, unknown): “[21, 22] propose modiﬁcations to feed-forward networks to include interpretable components, i.e. replacing usual activation functions by operators such as cos , sin , however these are hard to optimize and prone to numerical issues.”
Evaluating explorative prediction power of machine learning algorithms for materials discovery using k-fold forward cross-validation (2020, background): “Margius and Lampert [28] proposed a neural network approach to learn interpretable functions using an end-to-end differentiable feedforward neural network framework with efficient gradient-based training.”
Identifying Causal Structure in Large-Scale Kinetic Systems (2018, background): “…are proposed in Todorovski and Dzeroski [1997], Zembowicz and Zytkow [1992], Washio et al. [1999], while modern approaches even try end-to-end learning [Raissi, 2018, Martius and Lampert, 2016] and combine the identification of underlying differential equations from data with prediction.”