Reflexion: language agents with verbal reinforcement learning

By Noah Shinn, Federico Cassano, Beck Labash, A. Gopinath, Karthik Narasimhan, Shunyu Yao. 2023

4,400 citations recorded by the source corpus.

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

Subfields: Cognitive Neuroscience, Cultural Studies, Developmental and Educational Psychology, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

  1. Armando Solar-Lezama
  2. Xinyun Chen
  3. Pushmeet Kohli
  4. Mohit Bansal
  5. Noah D. Goodman
  6. Nitesh V. Chawla
  7. Karthik Narasimhan
  8. Graham Neubig

In Cultural Studies

  1. Noah D. Goodman
  2. Jie Huang
  3. Graham Neubig
  4. Marinka Zitnik
  5. Tomer D. Ullman

In Developmental and Educational Psychology

  1. Xinyun Chen
  2. Pushmeet Kohli
  3. Mohit Bansal
  4. Noah D. Goodman
  5. Karthik Narasimhan
  6. Jie Huang
  7. Y. Liu
  8. Graham Neubig

In Natural Language Processing

  1. Mohit Bansal
  2. Graham Neubig
  3. Zhiyuan Liu
  4. Maosong Sun
  5. Jianfeng Gao
  6. Luke S. Zettlemoyer
  7. Richard Socher
  8. Caiming Xiong

In Reinforcement Learning

  1. Shunyu Yao
  2. Xinyun Chen
  3. Karthik Narasimhan
  4. Graham Neubig
  5. Jianfeng Gao
  6. Mohit Bansal
  7. Richard Socher
  8. Armando Solar-Lezama

Citations in the corpus