Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text

By Felix Hill, Sona Mokra, Nathaniel Wong, Tim Harley. 2020

92 citations recorded by the source corpus.

Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of multi-goal motor policies via deep RL from scratch requires many episodes of experience. Consequently, instruction-following with deep RL typically involves language generated from templates (by an environment simulator), which does not reflect the varied or ambiguous expressions of real users. Here, we propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions. By applying our method with a state-of-the-art pre-trained text-based language model (BERT), on tasks requiring agents to identify and position everyday objects relative to other objects in a naturalistic 3D simulated room, we demonstrate substantially-above-chance zero-shot transfer from synthetic template commands to natural instructions given by humans. Our approach is a general recipe for training any deep RL-based system to interface with human users, and bridges the gap between two research directions of notable recent success: agent-centric motor behavior and text-based representation learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “While prior works have studied how pre-trained language embeddings can improve generalization to new instructions [38, 22, 21] and to new low-level tasks [13], we extract much more substantial knowledge from LLMs by grounding them within the robot’s affordances.”
Pre-Trained Language Models for Interactive Decision-Making (2022, background): “Some of the most successful applications of pre-training lie at the boundary of natural language processing and other domains, as in instruction following (Hill et al., 2020) and language-guided image retrieval (Lu et al.”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “…power of neural architectures, various works looked into creating instruction-following agents that can perform manipulation (Lynch & Sermanet, 2020; 2021), navigation (Fried et al., 2018; Wang et al., 2019; Majumdar et al., 2020), or both (Suglia et al., 2021; Hill et al., 2020; Fu et al., 2019).”
LILA: Language-Informed Latent Actions (2021, methodology): “…sought to combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], perform imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for reinforcement learning [12, 37, 38].”
Aligning AI With Shared Human Values (2020, background): “When more ethical knowledge is loaded during model pretraining, the representations may enable a regularizer for selecting good from bad actions in open-world or reinforcement learning settings (Hausknecht et al., 2019; Hill et al., 2020), or they may be used to steer text generated by a chatbot.”
Implicit Representations of Meaning in Neural Language Models (2021, methodology): “…deal of past research uses language modeling as a pretraining scheme for more conventional (supervised) semantics tasks in NLP. LM pretraining is useful for semantic parsing (Einolghozati et al., 2019), instruction following (Hill et al., 2020), and even image retrieval (Ilharco et al., 2021).”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Other works have incorporated pre-trained language models, often either as an instruction encoder [63, 64, 43, 41, 44, 1, 65] or for high-level planning [66, 19, 17, 67, 68, 69].”
Intra-agent speech permits zero-shot task acquisition (2022, background): “Recently, there have been efforts to demonstrate zero-shot task acquisition using a variety of means including with the use of pretrained models for language encoders [36] or visual-language models for zero-shot navigation [37].”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “Additionally, we observe that by virtue of leveraging pretrained language models our learned reward is capable of generalizing from scripted language instructions to unseen natural language zero-shot, suggesting that knowledge in pretrained language models can enable more efficient learning of groun”
Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning (2023, unknown): “, 2022] and Inner Monologue [Huang et al., 2022b] used LLMs as high-level planners in robotics setups. Because their LLM is not directly used as agent policy for low-level actions and is not grounded using its interactions with the environment, Ahn et al. [2022] had to use an external affordance fun”
Word meaning in minds and machines (2020, methodology): “Recent work has begun exploring ways of utilizing pretrained language models for instruction-following tasks, enriching the underlying word representations and facilitating transfer to novel synonyms and more natural forms of instruction (Hill, Mokra, et al., 2020).”
Language Grounding with 3D Objects (2021, background): “Agents can perform information-seeking behaviors on objects to improve their world [15, 16, 17] and language understanding [18, 19, 20].”
ELLA: Exploration through Learned Language Abstraction (2021, background): “sample-inefficient but has the potential to flexibly deal with complex environments (Hill et al., 2020).”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “The instruction-following literature includes several examples of language-based learning agents, centered on navigation [31], [32], [33], [10], [9], [34], [12], [35] or simple manipulation tasks [11], [36], [22].”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “Learning generalizable robot policies requires developing methods that can utilize X-embodiment data, tapping into datasets from many labs, robots, and settings.”
HIGhER: Improving instruction following with Hindsight Generation for Experience Replay (2020, methodology): “Recent works using pretrained language model [14, 23] could also complement HIGhER both in the generator and in the”