Semantic Exploration from Language Abstractions and Pretrained Representations

By Allison C. Tam, Neil C. Rabinowitz, Andrew Kyle Lampinen, Nicholas A. Roy, Stephanie C. Y. Chan, DJ Strouse, Jane X. Wang, Andrea Banino, Felix Hill. 2022

84 citations recorded by the source corpus.

Effective exploration is a challenge in reinforcement learning (RL). Novelty-based exploration methods can suffer in high-dimensional state spaces, such as continuous partially-observable 3D environments. We address this challenge by defining novelty using semantically meaningful state abstractions, which can be found in learned representations shaped by natural language. In particular, we evaluate vision-language representations, pretrained on natural image captioning datasets. We show that these pretrained representations drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by considering the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach in two very different task domains -- one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Training Language Models with Language Feedback (2022, background): “, 2022, inter alia), drive exploration (Tam et al., 2022), infer reward functions (Lin et al.”
Training Language Models with Natural Language Feedback (2022, background): “, 2022, inter alia), drive exploration (Tam et al., 2022), infer reward functions (Lin et al.”
Training Language Models with Language Feedback at Scale (2023, background): “…various purposes (see Luketina et al., 2019, for an overview), such as specifying tasks (“instruction following” , inter alia ) driving exploration (Tam et al., 2022), inferring reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia ), and training a model via…”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, unknown): “A related line of works has also explored using LLMs for reward specification in the context of reward design [76] and exploration in reinforcement learning [77–80], and human preference learning [81].”
Improving Intrinsic Exploration with Language Abstractions (2022, methodology): “More general extensions include better understanding when and why language semantics beneﬁts exploration (Section 7.2) and using pretrained models to imbue semantics into the models beforehand [49].”
Tell me why! Explanations support learning relational and causal structure (2021, methodology): “, 2017), to target exploration (Goyal et al., 2019; Watkins et al., 2021; Mu et al., 2022; Tam et al., 2022), or as an abstraction to structure hierarchical policies (Jiang et al.”
Learning from Natural Language Feedback (2024, background): “Language is also used to drive meaningful exploration (Tam et al., 2022) or to infer rewards in a certain context (Lin et al.”
History Compression via Language Models in Reinforcement Learning (2022, background): “, 2022) or pretrained multimodal models (Tam et al., 2022) for abstraction to enhance exploration, and in the offline RL setting for sequence modeling (Reid et al.”
Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (2022, background): “95,96 A second ability is systematic generalization .”
Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling (2023, methodology): “Some preliminary methods for generating state descriptions from images are used by Tam et al. (2022), but this remains an open area of research.”