Pre-Trained Language Models for Interactive Decision-Making

By Shuang Li, Xavier Puig, Yilun Du, Clinton Jia Wang, Ekin Akyürek, Antonio Torralba, Jacob Andreas, Igor Mordatch. 2022

334 citations recorded by the source corpus.

Language model (LM) pre-training is useful in many language processing tasks. But can pre-trained LMs be further leveraged for more general machine learning problems? We propose an approach for using LMs to scaffold learning and generalization in general sequential decision-making problems. In this approach, goals and observations are represented as a sequence of embeddings, and a policy network initialized with a pre-trained LM predicts the next action. We demonstrate that this framework enables effective combinatorial generalization across different environments and supervisory modalities. We begin by assuming access to a set of expert demonstrations, and show that initializing policies with LMs and fine-tuning them via behavior cloning improves task completion rates by 43.6% in the VirtualHome environment. Next, we integrate an active data gathering procedure in which agents iteratively interact with the environment, relabel past"failed"experiences with new goals, and update their policies in a self-supervised loop. Active data gathering further improves combinatorial generalization, outperforming the best baseline by 25.1%. Finally, we explain these results by investigating three possible factors underlying the effectiveness of the LM-based policy. We find that sequential input representations (vs. fixed-dimensional feature vectors) and LM-based weight initialization are both important for generalization. Surprisingly, however, the format of the policy inputs encoding (e.g. as a natural language string vs. an arbitrary sequential encoding) has little influence. Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.

Subfields: Artificial Intelligence, Prompting & Reasoning, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

In Reinforcement Learning

Citations in the corpus

ReAct: Synergizing Reasoning and Acting in Language Models (2022, neutral): “We also note that leveraging language as semantically-rich inputs in the process of interactive decision making has been shown to be successful under other settings (Abramson et al., 2020; Karamcheti et al., 2021; Huang et al., 2022a; Li et al., 2022).”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Another approach finetunes language models with interactive data, such as rewards or ranking feedback of the interaction [3, 39, 40].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Various prior works have explored using language as a space for planning [51, 52, 53, 20, 54, 21].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “Concurrent work by Li et al. (2022) validates similar hypothesis that LMs contain rich actionable knowledge.”
PaLM-E: An Embodied Multimodal Language Model (2023, neutral): “In LID (Li et al., 2022), this constraint is relaxed and LLM parameters are finetuned to produce a planning network for generating high-level instructions.”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “Recent approaches exploring foundation models for robotics have demonstrated impressive open-set interaction abilities [36, 37, 38, 39, 40].”
The alignment problem from a deep learning perspective (2022, background): “…such as model-based RL and planning [Sutton and Barto, 2018] (with learned reward functions), goal-conditioned sequence modeling [Chen et al., 2021, Li et al., 2022, Schmidhuber, 2020], or RL on rewards learned via inverse RL [Ng and Russell, 2000]—however, these are beyond our current scope.”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “Pretrained language models can also provide useful initializations for training policies to imitate offline trajectories [42, 27].”
Composing Ensembles of Pre-trained Models via Iterative Consensus (2022, positive): “Large pre-trained models have been widely applied to solve decisionmaking tasks, such as learning general purpose policies (Reed et al., 2022; Li et al., 2022; Shridhar et al., 2022), making planners (Huang et al.”
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning (2023, methodology): “Along with using off-the-shelf LLMs, there are works that either fine-tune LLMs [55, 38] or train sequence models [62, 27, 7, 43] for sequential decision making tasks.”
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models (2022, background): “Prompting LLMs to generate text useful for robot task planning is a nascent topic [32], [33], [34], [2], [4], [1].”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “Similarly, it would be interesting to see if VPT could beneﬁt from pretraining its attention layers with a language modeling task as in Li et al. 68 and Reid et al. 69 Loss was not consistently correlated with downstream evaluation metrics (Sec.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Thus, starting with BERT [103], it has become a standard practice to pretrain and then fine-tunes a large LM for plenty of downstream tasks in lieu of training a model from scratch for a specific task [103, 19, 4, 105, 112].”
Tool Learning with Foundation Models (2023, neutral): “For instance, Li et al. (2022) utilize foundation models as policy networks, whose input is the tokenized environment observations, the original goals, and action history.”