Beating Atari with Natural Language Guided Reinforcement Learning

By Russell Kaplan, Chris Sauer, A. Sosa. 2017

73 citations recorded by the source corpus.

We introduce the first deep reinforcement learning agent that learns to beat Atari games with the aid of natural language instructions. The agent uses a multimodal embedding between environment observations and natural language to self-monitor progress through a list of English instructions, granting itself reward for completing instructions in addition to increasing the game score. Our agent significantly outperforms Deep Q-Networks (DQNs), Asynchronous Advantage Actor-Critic (A3C) agents, and the best agents posted to OpenAI Gym on what is often considered the hardest Atari 2600 environment: Montezuma's Revenge.

Subfields: LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In LLM Agents & Tool Use

In Reinforcement Learning

Citations in the corpus

Training Language Models with Language Feedback (2022, background): “, 2017, inter alia), and train the model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017), reward shaping (Goyal et al.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “Lastly, there are a number of prior works that study how language can guide or improve reinforcement learning [37, 19, 30, 6, 20, 12, 10].”
Learning Rewards from Linguistic Feedback (2020, background): “…in machine learning uses language to improve sample efﬁciency: to shape rewards (Maclin and Shavlik 1994; Kuhlmann et al. 2004), often via subgoals (Kaplan, Sauer, and Sosa 2017; Williams et al. 2018; Chevalier-Boisvert et al. 2018; Goyal, Niekum, and Mooney 2019; Bahdanau et al. 2019; Zhou and…”
Tell me why! - Explanations support learning of relational and causal structure (2021, methodology): “Language in RL: Language is used broadly in RL, whether as instructions (e.g. Hermann et al., 2017; Kaplan et al., 2017), to target exploration (Goyal et al., 2019), or as an abstraction to structure hierarchical policies (Jiang et al., 2019).”
Training Language Models with Language Feedback at Scale (2023, methodology): “Korbak et al. (2022) show that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by a reward function.”
Teaching Machines to Describe Images via Natural Language Feedback (2017, methodology): “Parallel to our work, [7] exploits textual advice to improve training time of the A3C algorithm in playing an Atari game.”
Continual and Multi-task Reinforcement Learning With Shared Episodic Memory (2019, methodology): “In that case sub-task networks share part of their layers and the representation of the agent’s current task is often fed in as additional input to the network (Florensa et al., 2017; Dosovitskiy & Koltun, 2016; Kaplan et al., 2017).”
Using Natural Language for Reward Shaping in Reinforcement Learning (2019, background): “, 2012a] and [Kaplan et al., 2017] use natural language to do high-level planning.”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, background): “Kuhlmann et al. (2004) generate rules in a custom language from a set of natural language instructions.”
Training Language Models with Natural Language Feedback (2022, background): “…Chaplot et al., 2017; Mahmoudieh et al., 2022; Ouyang et al., 2022, inter alia ), drive exploration (Tam et al., 2022), infer reward functions (Lin et al., 2022), and train the model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017) or reward shaping (Goyal et al., 2019).”
PERCEPTUAL GOAL SPECIFICATIONS FOR REINFORCEMENT LEARNING (2017, background): “Recently, an approach learned correspondences between written instructions and visual states in Atari games [30].”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “We then observed the use of sentence embeddings (usually by an LSTM or GRU) combined with other sources such as image embeddings (Misra et al., 2017; Hermann et al., 2017; Kaplan et al., 2017; Fu et al., 2019) or instruction memory (Oh et al., 2017).”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “In the reinforcement learning setting, works have adopted a range of strategies, from language-conditioned reinforcement learning while leveraging environment rewards [21, 41, 42, 43, 44, 45] to using language as a reward bonus to densify the environment reward and aid in exploration [46, 47, 48, 49”
An Introduction to Deep Reinforcement Learning (2018, unknown): “One line of work suggests using natural language to guide the agent by providing exploration bonuses when an instruction is correctly executed (Kaplan et al., 2017).”
Learning from Natural Language Feedback (2024, background): “Other work uses language to train the model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017) or reward shaping (Goyal et al., 2019).”
Tell me why! Explanations support learning relational and causal structure (2021, methodology): “Language in RL: Language is used broadly in RL, whether as instructions (e.g. Hermann et al., 2017; Kaplan et al., 2017), to target exploration (Goyal et al.”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, methodology): “edge can also be incorporated through natural language interfaces[99].”