Improving Intrinsic Exploration with Language Abstractions

By Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah D. Goodman, Tim Rocktaschel, Edward Grefenstette. 2022

78 citations recorded by the source corpus.

Reinforcement learning (RL) agents are particularly hard to train when rewards are sparse. One common solution is to use intrinsic rewards to encourage agents to explore their environment. However, recent intrinsic exploration methods often use state-based novelty measures which reward low-level exploration and may not scale to domains requiring more abstract skills. Instead, we explore natural language as a general medium for highlighting relevant abstractions in an environment. Unlike previous work, we evaluate whether language can improve over existing exploration methods by directly extending (and comparing to) competitive intrinsic exploration baselines: AMIGo (Campero et al., 2021) and NovelD (Zhang et al., 2021). These language-based variants outperform their non-linguistic forms by 47-85% across 13 challenging tasks from the MiniGrid and MiniHack environment suites.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Yejin Choi
3. Hannaneh Hajishirzi
4. Sergey Levine
5. Luke S. Zettlemoyer
6. Yoshua Bengio
7. Jianfeng Gao
8. Kyunghyun Cho

In Natural Language Processing

1. Hannaneh Hajishirzi
2. Yejin Choi
3. Mohit Bansal
4. Sameer Singh
5. Luke S. Zettlemoyer
6. Tim Rocktaschel
7. Jonathan Berant
8. Noah A. Smith

In Reinforcement Learning

1. Felix Hill
2. Hannaneh Hajishirzi
3. Yejin Choi
4. Tim Rocktaschel
5. Andrew Kyle Lampinen
6. Markus Holzleitner
7. Hamid Eghbalzadeh
8. Jack Parker-Holder

Citations in the corpus

• History Compression via Language Models in Reinforcement Learning (2022, methodology): “, 2021), using language instructions (Mu et al., 2022) or pretrained multimodal models (Tam et al.”
• Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “[34] shows how language, in the form of hand-crafted BabyAI annotations, can help improve exploration in 2D environments.”
• Tell me why! Explanations support learning relational and causal structure (2021, methodology): “, 2017), to target exploration (Goyal et al., 2019; Watkins et al., 2021; Mu et al., 2022; Tam et al., 2022), or as an abstraction to structure hierarchical policies (Jiang et al.”
• Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Our work also relates to curriculum learning (Matiisen et al., 2020; Porte-las et al., 2019; Sukhbaatar et al., 2018; OpenAI et al., 2021; Campero et al., 2021; Fang et al., 2021; Mu et al., 2022), with the key difference that these methods typically have a specific downstream goal or task in mind.”
• Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling (2023, background): “Tam et al. (2022) and Mu et al. (2022) recently demonstrated that language is a meaningful state abstraction when used for exploration.”