Reflexion: language agents with verbal reinforcement learning
By Noah Shinn, Federico Cassano, Beck Labash, A. Gopinath, Karthik Narasimhan, Shunyu Yao. 2023
4,400 citations recorded by the source corpus.
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.
Subfields: Cognitive Neuroscience, Cultural Studies, Developmental and Educational Psychology, Natural Language Processing, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
- Armando Solar-Lezama
- Xinyun Chen
- Pushmeet Kohli
- Mohit Bansal
- Noah D. Goodman
- Nitesh V. Chawla
- Karthik Narasimhan
- Graham Neubig
In Cultural Studies
In Developmental and Educational Psychology
- Xinyun Chen
- Pushmeet Kohli
- Mohit Bansal
- Noah D. Goodman
- Karthik Narasimhan
- Jie Huang
- Y. Liu
- Graham Neubig
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Zhiyuan Liu
- Maosong Sun
- Jianfeng Gao
- Luke S. Zettlemoyer
- Richard Socher
- Caiming Xiong
In Reinforcement Learning
- Shunyu Yao
- Xinyun Chen
- Karthik Narasimhan
- Graham Neubig
- Jianfeng Gao
- Mohit Bansal
- Richard Socher
- Armando Solar-Lezama
Citations in the corpus
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024, methodology): “The self-repair scenario is based on previous works that tested the self-repair capabilities of LLMs (Olausson et al., 2023; Shinn et al., 2023; Chen et al., 2023).”
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024, background): “The environment can help unlock the capabilities like self-debugging [15] and self-reflection [53].”
- Large Language Models Cannot Self-Correct Reasoning Yet (2023, methodology): “For HotpotQA, we use the same prompt as Shinn et al. (2023).”
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (2023, background): “Our DFSDT extends Reflexion to a more general method by allowing LLMs to assess different reasoning paths and select the most promising one.”
- Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, background): “Some works have investigated how models can learn from their reasoning mistakes in-context (Shinn et al., 2023; Madaan et al., 2023; Zhang et al., 2024; Liu et al., 2023”
- Cognitive Architectures for Language Agents (2023, methodology): “B : Language agents (Ahn et al., 2022; Huang et al., 2022c) place the LLM in a direct feedback loop with the external environment by transforming observations into text and using the LLM to choose actions.”
- ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution (2024, positive): “Self-Reflections of LLMs. Shinn et al. (2023) propose to reinforce language agents via linguistic feedback, which is subsequently harnessed for various tasks (Madaan et al., 2023b; Wang et al., 2023e).”
- VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, positive): “Their effectiveness is further amplified with methods such as chain-of-thought reasoning or self-reflection [41,52].”
- rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, positive): “The community has actively explored various approaches, including self-correction [Huang et al., 2023, Kumar et al., 2024], self-reflection [Renze and Guven, 2024, Shinn et al., 2024], to explicitly train or prompt LLMs to develop such capability.”
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents (2024, positive): “For example, thanks to OpenHands’s extensible architecture, Auto Eval & Refine Pan et al. (2024), an agent retry-on-error strategy with Reflex-ion Shinn et al. (2024) prompts and task completion reward models, will be integrated as an optional component attached to our browsing agent.”
- A survey on large language model based autonomous agents (2023, neutral): “Common metrics include success rate [156, 151, 125, 90], reward/score [156, 151, 99], coverage [161], and accuracy [113, 1, 61].”
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (2023, positive): “Due to these limitations, even the multi-agent systems in LangChain (e.g., re-implementation of CAMEL) are not based on LangChain Agents but are implemented from scratch.”
- A Survey on Large Language Models for Code Generation (2024, neutral): “CodeBLEU combines these elements (including weighted n-gram match, syntactic AST match, and semantic data-flow match) into a comprehensive evaluation metric.”
- A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “For example, Shinn et al. [138] incorporate both in-trial and cross-trial information to build the memory module for enhancing the agent’s reasoning capability.”
- Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, methodology): “We evaluate internal reasoning strategies by removing actions and observations from the context, corresponding to CoT (Wei et al., 2022) and its variants, CoT-SC (Wang et al., 2022), ToT (Yao et al., 2023a), and RAP (Hao et al., 2023).”
- ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, background): “…veri-fiers (Cobbe et al., 2021), self-collaboration (Wang et al., 2023c; Schick et al., 2022; Li et al., 2023a; Feng et al., 2024), self-reflection (Shinn et al., 2023; Madaan et al., 2023; Wang and Zhao, 2023; Yao et al., 2023b), improved math reasoning (Yue et al., 2023; Luo et al., 2023) and…”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “Reflexion [38] proposes a novel verbal reinforcement learning framework, replacing weight updates in RL models with linguistic feedback.”