Self-Educated Language Agent with Hindsight Experience Replay for Instruction Following

By Geoffrey Cideron, Mathieu Seurin, Florian Strub, Olivier Pietquin. 2019

38 citations recorded by the source corpus.

Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. These properties make it a natural fit to guide the training of interactive agents as it could ease recurrent challenges in Reinforcement Learning such as sample complexity, generalization, or multi-tasking. Yet, it remains an open-problem to relate language and RL in even simple instruction following scenarios. Current methods rely on expert demonstrations, auxiliary losses, or inductive biases in neural architectures. In this paper, we propose an orthogonal approach called Textual Hindsight Experience Replay (THER) that extends the Hindsight Experience Replay approach to the language setting. Whenever the agent does not fulfill its instruction, THER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, THER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER. We observe that this simple idea also initiates a learning synergy between language acquisition and policy learning on instruction following tasks in the BabyAI environment.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “A large number of prior works have learned language-conditioned behavior via imitation learning [51, 22, 20, 13, 26, 37] or reinforcement learning [52, 53, 49, 54, 55, 56, 21, 41].”
Language Models are Few-Shot Butlers (2021, neutral): “In ALFWorld, learning such a mapping is highly non-trivial and constitutes another research direction for extending existing methods (Cideron et al., 2019) to this environment.”
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, background): “Early works with templated language goals rely on environment simulators to provide hindsight labels [10, 25], and more recently [12, 39] use learned models to predict templated language instructions.”
Grounding Language to Autonomously-Acquired Skills via Goal Generation (2021, unknown): “…deﬁne instruction following agents that receive external instructions and rewards (Hermann et al., 2017; Chan et al., 2019; Bahdanau et al., 2018; Cideron et al., 2019; Jiang et al., 2019; Fu et al., 2019), except the IMAGINE approach which introduced intrinsically motivated agents able to set…”
Learning Rewards from Linguistic Feedback (2020, background): “A somewhat smaller body of work investigates such open-ended language: to correct captioning models (Ling and Fidler 2017), capture environmental characteristics (Narasimhan, Barzilay, and Jaakkola 2018), or improve hindsight replay (Cideron et al. 2019).”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, methodology): “These works then use the learned reward to optimize policies through online RL, often using the agents own online experience to train the reward [22, 51].”
Explainability in Deep Reinforcement Learning (2020, methodology): “Transparent algorithms are well known and used in standard Machine Learning (e.g., linear regression, decision trees or rule-based systems).”