Learning Compositional Neural Programs with Recursive Tree Search and Planning

By Thomas Pierrot, Guillaume Ligner, Scott E. Reed, Olivier Sigaud, Nicolas Perrin, Alexandre Laterre, David Kas, Karim Beguir, Nando de Freitas. 2019

43 citations recorded by the source corpus.

We propose a novel reinforcement learning algorithm, AlphaNPI, that incorporates the strengths of Neural Programmer-Interpreters (NPI) and AlphaZero. NPI contributes structural biases in the form of modularity, hierarchy and recursion, which are helpful to reduce sample complexity, improve generalization and increase interpretability. AlphaZero contributes powerful neural network guided search algorithms, which we augment with recursion. AlphaNPI only assumes a hierarchical program specification with sparse rewards: 1 when the program execution satisfies the specification, and 0 otherwise. Using this specification, AlphaNPI is able to train NPI models effectively with RL for the first time, completely eliminating the need for strong supervision in the form of execution traces. The experiments show that AlphaNPI can sort as well as previous strongly supervised NPI variants. The AlphaNPI agent is also trained on a Tower of Hanoi puzzle with two disks and is shown to generalize to puzzles with an arbitrary number of disk

Subfields: Artificial Intelligence, Control and Systems Engineering, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Evaluating Large Language Models Trained on Code (2021, methodology): “More recent approaches like the Neural Program Interpreter (Reed & de Freitas, 2016; Shin et al., 2018; Pierrot et al., 2021) and”
Meta-learning curiosity algorithms (2020, background): “…al., 2017; Gaier & Ha, 2019), searching over programs to solve games (Wilson et al., 2018; Kelly & Heywood, 2017; Silver et al., 2019) and to optimize neural networks (Bengio et al., 1995; Bello et al., 2017), and neural networks that learn programs (Reed & De Freitas, 2015; Pierrot et al., 2019).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “Existing works have used these methods in varied application domains: toy programming tasks (Bošnjak et al., 2017; Cai et al., 2017; Bunel et al., 2018; Pierrot et al., 2019), VQA (Andreas et al.”
Explainability in Deep Reinforcement Learning (2020, unknown): “While improving training time (compared to more traditional approaches [45]), this method also allows for better understanding of the agent’s behaviour as it learns according to human expert actions [99].”
Compositional Generalization via Neural-Symbolic Stack Machines (2020, background): “Another class of approaches incorporate a symbolic machine into the neural network [38, 6, 49, 8, 47, 37], which enables length generalization either with training supervision on execution traces [6], or well-designed curriculum learning schemes [8, 47, 37].”
Evolving Reinforcement Learning Algorithms (2021, background): “Neural Programmer-Interpreters (Reed & De Fre-itas, 2015; Pierrot et al., 2019) overcome the environment generalization challenge by learning hierarchical neural programs with domain-speciﬁc encoders for different environments.”