End-to-end symbolic regression with transformers
By Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample, Franccois Charton. Neural Information Processing Systems, 2022
295 citations recorded by the source corpus.
Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the"skeleton"of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function. The dominant approach is genetic programming, which evolves candidates by iterating this subroutine a large number of times. Neural networks have recently been tasked to predict the correct skeleton in a single try, but remain much less powerful. In this paper, we challenge this two-step procedure, and task a Transformer to directly predict the full mathematical expression, constants included. One can subsequently refine the predicted constants by feeding them to the non-convex optimizer as an informed initialization. We present ablations to show that this end-to-end approach yields better results, sometimes even without the refinement step. We evaluate our model on problems from the SRBench benchmark and show that our model approaches the performance of state-of-the-art genetic programming with several orders of magnitude faster inference.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/aee15eb44604f4ad04d0c8ca7d41c87d03db5a39
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jure Leskovec
- William L. Hamilton
- Jian Tang
- Marinka Zitnik
- Tie-Yan Liu
- Percy Liang
- Yoshua Bengio
- Zhangyang Wang
Citations in the corpus
- Scientific discovery in the age of artificial intelligence (2023, background): “…dominate natural language processing 37,99 and have been successfully applied to a range of problems, including earthquake signal detection 119 , DNA and protein sequence modelling 10,120 , modelling the effect of sequence variation on biological function 100,121 , and symbolic regression 122 .”
- Interpretable scientific discovery with symbolic regression: a review (2022, unknown): “More recently, SR has been addressed with deep learning-based methods [27, 28, 19, 29, 30, 31, 32, 33] which leverage neural networks (NNs) to learn accurate symbolic models.”
- Learning to Optimize: A Primer and A Benchmark (2021, background): “[102] provides a Transformer-based endto-end solution for symbolic L2O that directly predicts the full mathematical expression,”