TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning

By Gregory Farquhar, Tim Rocktaschel, Maximilian Igl, Shimon Whiteson. 2017

77 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “1As a concrete example of what a neural network optimizer might look like, consider TreeQN.[2] TreeQN, as described in Farquhar et al.”
Combinatorial optimization and reasoning with graph neural networks (2021, methodology): “Additionally, the model performs strongly in the low-data regime against ATreeC (Farquhar et al., 2018) that resorts to predicting scalar values in every node of the inferred local MDP, so that VI-style rules can be directly applied exactly according to the predictions above.”
Neural algorithmic reasoning (2021, methodology): “Besides improved data eﬃciency over more traditional approaches to RL, it also compared favourably against ATreeC [Farquhar et al., 2017], which attempts to directly apply VI in a neural pipeline, thus encountering the algorithmic bottleneck problem in low-data regimes.”
Universal Planning Networks (2018, background): “Concurrently, a few recent efforts have been developed to embed differentiable planning procedures in computation graphs (Guez et al., 2018; Pereira et al., 2018; Farquhar et al., 2017).”
QUOTA: The Quantile Option Architecture for Reinforcement Learning (2018, methodology): “All the hyper-parameter values above were based on an n -step Q-learning baseline with synchronous workers from Farquhar et al. (2018), and all our implementations inherited these hyper-parameter values.”
Combined Reinforcement Learning via Abstract Representations (2018, result): “As compared to treeQn (Farquhar et al. 2017), the learning of the model is explicit and we show how that approach allows recovering a sufﬁcient interpretable low-dimensional representation of the environment, even in the absence of model-free objectives.”
Decoupling Dynamics and Reward for Transfer Learning (2018, methodology): “Results in Fig.6 show that both our method (DDR planning) and ATreeC achieve comparable performance and do better than A3C, especially in the stochastic environments.”
Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning (2019, background): “…presenting unique challenges for policy learning: • Planning Horizon Atari environments requiring temporally extended planning strategies for success, as in Frost-bite, still present challenges to deep reinforcement learning algorithms (Jiang et al., 2015; Lake et al., 2017; Farquhar et al., 2018).”
The Value Equivalence Principle for Model-Based Reinforcement Learning (2020, background): “Algorithm Operator T̃ Policies Π Functions V Predictron [34] T̃ λvt None Value functions for pseudo-rewards VIN [40] T̃ nGvt vt Gvt Value function TreeQN [16] T̃ nGnvt vt G n vt Value function VPN [26] T̃ na1..anvt {a1, ..., an} ∼ πt Value function MuZero [33] T̃ na1...anvt {a1, ..., an} ∼ πt Distri”