HIGhER: Improving instruction following with Hindsight Generation for Experience Replay
By Geoffrey Cideron, Mathieu Seurin, Florian Strub, Olivier Pietquin. 2020
28 citations recorded by the source corpus.
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language understanding and reinforcement learning in even simple instruction following scenarios. This joint learning problem is alleviated through expert demonstrations, auxiliary losses, or neural inductive biases. In this paper, we propose an orthogonal approach called Hindsight Generation for Experience Replay (HIGhER) that extends the Hindsight Experience Replay approach to the language-conditioned policy setting. Whenever the agent does not fulfill its instruction, HIGhER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, HIGhER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER. We show the efficiency of our approach in the BabyAI environment, and demonstrate how it complements other instruction following methods.
Subfields: Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Reinforcement Learning
- Sergey Levine
- Pierre Sermanet
- Karol Hausman
- Jonathan Tompson
- Chelsea Finn
- Ted Xiao
- Pieter Abbeel
- Ayzaan Wahid
Citations in the corpus
- Language Models are Few-Shot Butlers (2021, background): “trivial and constitutes another research direction for extending existing methods (Cideron et al., 2020) to this environment.”
- Interactive Learning from Activity Description (2021, background): “Recently, several papers have proposed using language description feedback in the context of reinforcement learning (Jiang et al., 2019; Chan et al., 2019; Colas et al., 2020; Cideron et al., 2020).”
- Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, methodology): “This method has been applied to environments with goals represented as images [12], task IDs [27], and language instructions [11, 13, 25].”