From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

By Justin Fu, Anoop Korattikara Balan, Sergey Levine, Sergio Guadarrama. 2019

144 citations recorded by the source corpus.

Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a language-conditioned policy leads to poor performance.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Learning to Understand Goal Specifications by Modelling Reward (2018, background): “To our knowledge, ours and the concurrent work by Fu et al. (2018) are the ﬁrst works to showcase learning reward models for instructions from pixels directly.”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “At the same time, by leveraging the better representation power of neural architectures, a number of works have looked into creating instruction-following agents that can perform manipulation [29, 30], navigation [11, 54, 31], or both [49, 16, 12].”
LILA: Language-Informed Latent Actions (2021, background): “Other approaches use language in more nuanced ways, such as learning language-conditional reward functions directly [39, 40, 41], or within adaptive frameworks, where language is used to correct or define new behavior [42, 43].”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “When full demonstrations are available, the reward function can be learned using standard IRL methods like MaxEnt IRL [Ziebart et al. , 2008] as in [Fu et al. , 2019], or maximum likelihood IRL [Babes et al. , 2011] as in [MacGlashan et al. , 2015], who also learn a joint generative model of…”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “While language is an appealing choice as the abstraction for hierarchical RL, training a low-level policy to follow language instructions is highly non-trivial [20, 6] as it involves learning from binary rewards that indicate completion of the instruction.”
Grounding Language to Autonomously-Acquired Skills via Goal Generation (2021, unknown): “…receive external instructions and rewards (Hermann et al., 2017; Chan et al., 2019; Bahdanau et al., 2018; Cideron et al., 2019; Jiang et al., 2019; Fu et al., 2019), except the IMAGINE approach which introduced intrinsically motivated agents able to set their own goals and to imagine new ones…”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “The instruction-following literature includes several examples of language-based learning agents, centered on navigation [31], [32], [33], [10], [9], [34], [12], [35] or simple manipulation tasks [11], [36], [22].”
Learning to Generalize from Sparse and Underspecified Rewards (2019, background): “…environments where a learning agent is provided with a natural language input and asked to produce a sequence of actions for achieving a goal speciﬁed in the input text ( e.g., Long et al. (2016); Hermann et al. (2017); Chaplot et al. (2018); Fu et al. (2019); Chevalier-Boisvert et al. (2019)).”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “ptioning, text-to-image generation, and VQA. It is worth noting that there are other applications, such as text-based image retrieval [94], [164], [165], and visual-andlanguage navigation (VLN) [166]–[174], that we have not included in this paper due to space limitation. • Caption generation is a ta”
Zero-Shot Reward Specification via Grounded Natural Language (2022, background): “Several previous efforts train reward functions or policies that take natural language as input for goal description (Oh et al., 2017; Bahdanau et al., 2018; Zhou & Small, 2020; Goyal et al., 2020; Fu et al., 2019; Hermann et al., 2017; Shao et al., 2020).”
Rapid Task-Solving in Novel Environments (2020, methodology): “Future work may approach RTS problems with broader task distributions, e.g., by using generative grammars over tasks or by taking human input (Fu et al., 2019).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, unknown): “A body of work on using language feedback for reward inference and shaping might lessen this challenge [Fu et al., 2019, Goyal et al., 2019, Sumers et al., 2021, Zhou and Small, 2021, Lin et al., 2022, Yu et al., 2023], but thus far, these techniques have not been applied to LLMs.”
Language to Rewards for Robotic Skill Synthesis (2023, methodology): “To extend the end-to-end approaches to manipulation, a common approach is to utilize latent embeddings of language commands as multitask input context, and train with behavioral cloning [14, 32, 16], offline reinforcement learning [33], goal-conditioned reinforcement learning [34], or in a shared au”
HIGhER: Improving instruction following with Hindsight Generation for Experience Replay (2020, background): “Similarly, [17] directly learn a reward function and assess its transferability to new environments.”
Inferring Rewards from Language in Context (2022, methodology): “This model is representative of approaches from past work on language-conditioned reward learning (e.g., MacGlashan et al. 2015; Fu et al. 2019) that infer rewards purely from the actions that utterances refer to.”
An Overview of Natural Language State Representation for Reinforcement Learning (2020, methodology): “We then observed the use of sentence embeddings (usually by an LSTM or GRU) combined with other sources such as image embeddings (Misra et al., 2017; Hermann et al., 2017; Kaplan et al., 2017; Fu et al., 2019) or instruction memory (Oh et al., 2017).”
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning (2018, methodology): “Following the convention in prior work (Fu et al., 2019), we provide to the reward function the corresponding semantic images (bottom right).”
Effective and General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping (2019, background): “These can be both virtual—sometimes photo-realistic—[1, 2, 3, 4, 5] or physical environments [6, 7, 8].”
ELLA: Exploration through Learned Language Abstraction (2021, unknown): “, 2019) to inverse reinforcement learning (Fu et al., 2019).”