Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning
By Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, Pierre-Yves Oudeyer. 2023
272 citations recorded by the source corpus.
Recent works successfully leveraged Large Language Models'(LLM) abilities to capture abstract knowledge about world's physics to solve decision-making problems. Yet, the alignment between LLMs'knowledge and the environment can be wrong and limit functional competence due to lack of grounding. In this paper, we study an approach (named GLAM) to achieve this alignment through functional grounding: we consider an agent using an LLM as a policy that is progressively updated as the agent interacts with the environment, leveraging online Reinforcement Learning to improve its performance to solve goals. Using an interactive textual environment designed to study higher-level forms of functional grounding, and a set of spatial and navigation tasks, we study several scientific questions: 1) Can LLMs boost sample efficiency for online learning of various RL tasks? 2) How can it boost different forms of generalization? 3) What is the impact of online learning? We study these questions by functionally grounding several variants (size, architecture) of FLAN-T5.
Subfields: Artificial Intelligence, Cultural Studies, LLM Agents & Tool Use, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Timo Schick
- Kyunghyun Cho
- Yoshua Bengio
- Percy Liang
In LLM Agents & Tool Use
In Reinforcement Learning
- Ramakanth Pasunuru
- Stephen Marcus McAleer
- Pierre Baldi
- Yann LeCun
- Asli Celikyilmaz
- Geunwoo Kim
- Christoforos Nalmpantis
- Roberta Raileanu
Citations in the corpus
- Augmented Language Models: a Survey (2023, methodology): “Finally, the agent in Carta et al. (2023) uses a LM to generate action policies for text-based tasks.”
- A Survey of Large Language Models (2023, unknown): “, generating action plans in natural language to manipulate agents [380, 381].”
- Visual-RFT: Visual Reinforcement Fine-Tuning (2025, unknown): “While there has been significant research on using reinforcement learning to enhance LLMs during post-training [1, 3, 27, 30, 34, 35, 38, 44, 48, 62, 63], the progress for LVLMs has been slower.”
- Language Models can Solve Computer Tasks (2023, methodology): “[6] employed online RL fine-tuning to achieve functional grounding of LLMs in the BabyAI-Text environment.”