ELLA: Exploration through Learned Language Abstraction

By Suvir Mirchandani, Siddharth Karamcheti, Dorsa Sadigh. 2021

64 citations recorded by the source corpus.

Building agents capable of understanding language instructions is critical to effective and robust human-AI collaboration. Recent work focuses on training these agents via reinforcement learning in environments with synthetic language; however, instructions often define long-horizon, sparse-reward tasks, and learning policies requires many episodes of experience. We introduce ELLA: Exploration through Learned Language Abstraction, a reward shaping approach geared towards boosting sample efficiency in sparse reward environments by correlating high-level instructions with simpler low-level constituents. ELLA has two key elements: 1) A termination classifier that identifies when agents complete low-level instructions, and 2) A relevance classifier that correlates low-level instructions with success on high-level tasks. We learn the termination classifier offline from pairs of instructions and terminal states. Notably, in departure from prior work in language and abstraction, we learn the relevance classifier online, without relying on an explicit decomposition of high-level instructions to low-level instructions. On a suite of complex BabyAI environments with varying instruction complexities and reward sparsity, ELLA shows gains in sample efficiency relative to language-based shaping and traditional RL methods.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In LLM Agents & Tool Use

In Reinforcement Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Others have approached the problem through hierarchical learning [45, 46, 34, 47, 48, 49, 50].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “Recent works also use language as hierarchical abstractions to plan actions using imitation learning (Sharma et al., 2021) and to guide exploration in reinforcement learning (Mirchandani et al., 2021).”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “While these works also learn languageconditioned rewards, and in some cases also use discriminative techniques to learn the reward [22, 49], running language-conditioned online RL on a physical robot can be prohibitively time consuming.”
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models (2022, background): “Recent works have explored learning-based task & motion planning, using methods such as representation learning, hierarchical learning, language as planning space, learning compositional skills and more [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26].”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, methodology): “Language for exploration Some recent works have used language to guide agent learning, by either using language subgoals for exploration/planning or providing task-specific reward shaping [47, 33, 13, 19].”
Improving Intrinsic Exploration with Language Abstractions (2022, methodology): “Meanwhile, reward shaping and inverse RL methods [3, 5, 21–24, 31, 44, 51, 53] reward an agent for actions associated with linguistic descriptions, but again are primarily designed for instruction following, where an extrinsic goal is available to help shape intermediate rewards.”