Can Wikipedia Help Offline Reinforcement Learning?

By Machel Reid, Yutaro Yamada, Shixiang Shane Gu. 2022

104 citations recorded by the source corpus.

Fine-tuning reinforcement learning (RL) models has been challenging because of a lack of large scale off-the-shelf datasets as well as high variance in transferability among different environments. Recent work has looked at tackling offline RL from the perspective of sequence modeling with improved results as result of the introduction of the Transformer architecture. However, when the model is trained from scratch, it suffers from slow convergence speeds. In this paper, we look to take advantage of this formulation of reinforcement learning as sequence modeling and investigate the transferability of pre-trained sequence models on other domains (vision, language) when finetuned on offline RL tasks (control, games). To this end, we also propose techniques to improve transfer between these domains. Results show consistent performance gains in terms of both convergence speed and reward on a variety of environments, accelerating training by 3-6x and achieving state-of-the-art performance in a variety of tasks using Wikipedia-pretrained and GPT2 language models. We hope that this work not only brings light to the potentials of leveraging generic sequence modeling techniques and pre-trained models for RL, but also inspires future work on sharing knowledge between generative modeling tasks of completely different domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Another approach finetunes language models with interactive data, such as rewards or ranking feedback of the interaction [11, 44, 45].”
History Compression via Language Models in Reinforcement Learning (2022, background): “…et al., 2022), creating abstractions of human language instructions (Hill et al., 2021), using language instructions (Mu et al., 2022) or pretrained multimodal models (Tam et al., 2022) for abstraction to enhance exploration, and in the offline RL setting for sequence modeling (Reid et al., 2022).”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “Pretrained language models can also provide useful initializations for training policies to imitate ofﬂine trajectories [42, 27].”
Generating Images with Multimodal Language Models (2023, background): “Prior work showcased the effectiveness of grounding text-only LMs to images for vision-and-language tasks [56, 4, 29, 33, 31, 35], to embodied settings for robotics [3, 18], offline reinforcement learning [48], and more.”
Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022, background): “For example, recent work has shown that pre-training on language data was useful for ofﬂine reinforcement learning, but pre-training on vision data was not (Reid et al., 2022) – could this difference be due to the non-uniform, structured distribution of the language data?”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “Reid et al. [200] explore the potential of leveraging the sequence modeling formulation of reinforcement learning and examine the transferability of pretrained sequence models across different domains, such as vision and language.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, result): “This is also consistent with findings in offline RL, where additional data is often required to effectively scale the model (Reid et al., 2022).”
Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers (background): “For example, recent work has shown that pre-training on language data was useful for oﬄine reinforcement learning, but pre-training on vision data was not (Reid et al., 2022) – could this diﬀerence be due to the non-uniform, structured distribution of the language data?”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “Third, the agent’s architecture needs to be flexible enough to pursue any task in openended environments, and scalable enough to convert large-scale knowledge sources into actionable insights [15, 82].”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “Similarly, it would be interesting to see if VPT could beneﬁt from pretraining its attention layers with a language modeling task as in Li et al. 68 and Reid et al. 69 Loss was not consistently correlated with downstream evaluation metrics (Sec.”