Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains

By David Abel, Alekh Agarwal, Fernando Diaz, Akshay Krishnamurthy, Robert E. Schapire. 2016

43 citations recorded by the source corpus.

High-dimensional observations and complex real-world dynamics present major challenges in reinforcement learning for both function approximation and exploration. We address both of these challenges with two complementary techniques: First, we develop a gradient-boosting style, non-parametric function approximator for learning on $Q$-function residuals. And second, we propose an exploration strategy inspired by the principles of state abstraction and information acquisition under uncertainty. We demonstrate the empirical effectiveness of these techniques, first, as a preliminary check, on two standard tasks (Blackjack and $n$-Chain), and then on two much larger and more realistic tasks with high-dimensional observation spaces. Specifically, we introduce two benchmarks built within the game Minecraft where the observations are pixel arrays of the agent's visual field. A combination of our two algorithmic techniques performs competitively on the standard reinforcement-learning tasks while consistently and substantially outperforming baselines on the two tasks with high-dimensional observation spaces. The new function approximator, exploration strategy, and evaluation benchmarks are each of independent interest in the pursuit of reinforcement-learning methods that scale to real-world domains.

Subfields: Artificial Intelligence, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “While theoretical RL addresses variety of the trade-offs in the exploration-exploitation, this problem is still prominent in practical reinforcement learning research (Mnih et al., 2015; Abel et al., 2016; Azizzadenesheli et al., 2016b).”
Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks (2016, background): “Both the work on Atari games [19] and the recent Minecraft scenarios studied by researchers [1, 22] focus on the control of a single agent, with a ﬁxed, limited set of actions.”
#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning (2016, neutral): “Another method similar to hashing is proposed by [1], which clusters states and counts cluster centers instead of the true states, but this method has yet to be tested on standard exploration benchmark problems.”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, background): “Even though the theoretical RL addresses the exploration and exploitation trade-offs, these problems are still prominent in empirical reinforcement learning research [8], [27], [28].”
Supervision via competition: Robot adversaries for learning tasks (2016, background): “Much of the recent work in improving exploration for deep learned models has focused reducing uncertainty [22], [23] or is novelty seeking [24], [25].”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, unknown): “35, 53–”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “RL approaches [68,1,20,33,44,92,62,86] have been proposed to address navigation in synthetic layouts like mazes, arcade games and other visual environments [100,8,47,54,43,84].”
PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-Based Planning (2017, background): “Reinforcement learning has recently gained popularity in solving motion planning problems for systems with unknown dynamics [16], and has enabled robots to learn tasks that have been previously difficult or impossible [1], [6], [20].”
Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement (2017, background): “For instance, Project Malmo74 is a platform developed by Microsoft over the well-known video game sandbox Minecraft, where different tasks can be created and AI agents can be programmed for them (Johnson et al, 2016; Abel et al, 2016).”
Sample-Efficient Deep RL with Generative Adversarial Tree Search (2018, background): “While theoretical RL addresses the trade-off in explorationexploitation, this problem is still prominent in empirical reinforcement learning research (Mnih et al., 2015; Abel et al., 2016; Azizzadenesheli et al., 2016b).”
ORRB - OpenAI Remote Rendering Backend (2019, background): “Finally custom designed environments built on top of games and game engines like: Doom [7], Quake III Arena [4], Minecraft [1] or Unity [6, 5] offer a testbed for basic ML research.”
Boosted Fitted Q-Iteration (2017, background): “Recently, Abel et al. (2016) have empirically shown that a variant of boosting is able to learn near optimal policies in complex domains when coped with exploratory strategies.”
Two Body Problem: Collaborative Visual Task Completion (2019, background): “Consequently the task is well suited for an RL formulation, a perspective which has become popular recently [62, 1, 16, 17, 33, 42, 86, 59, 5, 8, 90, 25, 36, 91, 37].”
AndroidEnv: A Reinforcement Learning Platform for Android (2021, methodology): “Other methods leverage prior human demonstration or prior knowledge (Abel et al., 2015, 2016; Frazier and Riedl, 2019; Guss et al., 2019; Shu et al., 2017).”