Back to paper library
arXiv 2023NeurIPS 2023Reinforcement learningPaper notes

Reflexion

Remember for next time

Original paper

Reflexion: Language Agents with Verbal Reinforcement Learning

Paper authors

Noah Shinn · Federico Cassano · Edward Berman · Ashwin Gopinath · Karthik Narasimhan · Shunyu Yao

Publication
NeurIPS 2023Conference
First submitted to arXiv
arXiv:2303.11366
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{reinforcementlearningreflexion2023,
  title = {{Reflexion: Language Agents with Verbal Reinforcement Learning}},
  author = {Noah Shinn and Federico Cassano and Edward Berman and Ashwin Gopinath and Karthik Narasimhan and Shunyu Yao},
  year = {2023},
  eprint = {2303.11366},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.11366}
}
On this page

In the spring of 2023, research on using language models as agents suddenly took off. ReAct had models look things up while thinking, Toolformer had models call tools on their own, and others had models doing things in games, on web pages and in programming environments.

None of these agents learned anything from failing. Traditional reinforcement learning can learn from failure, but it takes a huge number of attempts and expensive fine-tuning, which isn't very practical for models with tens of billions of parameters. So agents at the time could only rely on the examples in their prompts. Get something wrong this time, and they'd get it wrong the same way next time.

Shinn et al. at Northeastern University in the US (later versions added Narasimhan and Yao from Princeton) took a different approach. After people make a mistake, they think back on where they went wrong and what to do next time. A model can do the same, writing its reflection down as text and putting it into the prompt for the next attempt. The weights don't change at all; everything it learns is written in text.

Defining the problem

An agent takes a series of actions in an environment and gets feedback at the end. That feedback is often very sparse, maybe just "success" or "failure," or a single score.

For traditional reinforcement learning, the hard part of learning from this kind of signal is credit assignment. Out of a chain of dozens of actions, which step went wrong? With just a 0 or a 1, that's hard to answer.

The authors want to know whether a language model can "amplify" this sparse signal into concrete advice on its own, something like "I picked up the wrong thing in step 3; next time I should go to the desk for the lamp first," and then use that text to guide the next attempt.

Solving the problem

Original Figure 1. How Reflexion works on three kinds of tasks. Decision making: the agent has to clean a pan and put it on the countertop, and tries to take the pan from the stove, but the pan isn't there; the rule-based check flags a hallucination, the reflection writes down that it "tried to pick up the pan in stoveburner 1" but "the pan was not in stoveburner 1," and the next time the agent looks in the right place first. Programming: write a function that checks whether parentheses are balanced; the unit tests it wrote itself fail, and the reflection points out that the code only checks that the numbers of left and right parentheses are equal, not their order, so next time it uses a correct check. Reasoning: asked for the profession two people share, the agent answers wrong the first time; the reflection points out that the mistake was assuming the two of them had the same multiple professions, and the next time it gets the answer right, novelist.
Original Figure 1, Shinn et al., from PDF page 3, CC BY 4.0.

Three models

Original Figure 2. (a) The Reflexion architecture. Inside the agent are three language models. The Actor acts in the environment based on short-term memory (this attempt's trajectory) and long-term memory (past reflections); the Evaluator gives internal feedback based on the trajectory; Self-reflection combines external and internal feedback, writes a reflection and stores it in long-term memory. (b) The algorithm: first generate a trajectory, evaluate it, write a reflection and store it in memory; then keep repeating "generate a trajectory, evaluate, write a reflection, add it to memory" until the evaluation passes or the maximum number of attempts is reached.
Original Figure 2, Shinn et al., from PDF page 4, CC BY 4.0.

Reflexion is made up of three roles, all of which can be the same language model with different prompts:

  • Actor: acts in the environment, producing text and actions. The authors tried two styles, chain of thought and ReAct, and add a block of memory to the prompt.
  • Evaluator: evaluates how this attempt went. Reasoning questions are judged by exact match with the reference answer; decision-making tasks use pre-written rules, or ask another language model to judge success or failure; programming problems run the unit tests the model wrote itself.
  • Self-reflection: given the success or failure signal, this attempt's trajectory and past memory, it writes a reflection that says what went wrong and what to do next time.

Two kinds of memory

The full trajectory of the current attempt is short-term memory, full of detail; the reflections written as text are long-term memory, more condensed, storing the lessons learned. On the next attempt, the Actor sees both kinds of memory.

Long-term memory can't grow forever, or it would exceed the prompt's length limit. The authors keep only the most recent few reflections: 3 for decision-making and reasoning tasks, 1 for programming tasks.

The loop

  1. The Actor makes one attempt and gets trajectory τ0\tau_0.
  2. The Evaluator gives a score r0r_0.
  3. Self-reflection writes reflection sr0sr_0 based on {τ0,r0}\{\tau_0, r_0\} and puts it into memory.
  4. Repeat: the Actor tries again with the memory, the attempt is evaluated, a new reflection is written and added to memory, until the Evaluator judges it a success or the attempts run out.

The authors call this "verbal" reinforcement learning. In traditional reinforcement learning, the policy is determined by weights; here the policy is "language model + memory," and the policy is updated by rewriting the text in memory. The authors also admit plainly that this approach depends on the language model's ability to evaluate itself, and there's no guarantee it will succeed.

Discussion

Decision making: AlfWorld

Original Figure 3. (a) Cumulative share of the 134 AlfWorld tasks solved, against the number of trials. ReAct alone rises from about 0.63 to about 0.75 and then stalls; ReAct + Reflexion climbs all the way to about 0.97 with rule-based judging, and to about 0.94 with GPT judging. (b) Share of failures by cause: with ReAct alone, failures from hallucination drop from about 0.32 to about 0.22 and stall there; with Reflexion added, hallucination falls to about 0.03 and inefficient planning to 0.
Original Figure 3, Shinn et al., from PDF page 6, CC BY 4.0.

AlfWorld is a text-based household environment where the agent has to complete tasks through text commands, such as finding a spatula hidden in a drawer, moving a knife to the cutting board, or chilling a tomato in the fridge. The authors follow ReAct's setup and test on 134 environments, with ReAct as the Actor.

This environment only tells the agent whether the task is done, so the Evaluator has to decide for itself when an attempt counts as a failure. The authors use two simple rules. If the same action is repeated more than 3 times with the same response, it's judged a hallucination; if more than 30 actions are taken in the environment, it's judged inefficient planning.

ReAct + Reflexion solves 130 of the 134 tasks, while ReAct alone stops improving after the 6th or 7th trial.

The right plot breaks failures down by cause. ReAct's most common failure is thinking it already has some item when it doesn't, then carrying out a long string of actions without ever finding its way back to the mistake; this kind of hallucination stays stuck at around 22%. Reflexion condenses a long failed trajectory into a few lines of reminders and avoids the mistake next time.

The appendix has a concrete example. The task is "examine the mug with the desklamp." The agent first finds the mug and picks it up, then goes looking for the desk lamp, and in the end can't use it. The reflection reads: my plan was to find the mug first and then the desk lamp, but the task is to examine the mug with the desk lamp, so I should have looked for the desk lamp first; I noticed the desk lamp is on desk 1, so next time I'll go straight to desk 1. On the second attempt, the agent finishes in three actions.

Reasoning: HotPotQA

Original Figure 4. Share of 100 HotPotQA questions solved, against the number of trials. (a) Chain of thought and ReAct alone both sit around 0.3 and don't change with more trials; with Reflexion added, ReAct rises to about 0.53 and chain of thought to about 0.4. (b) Chain of thought given the correct context paragraphs is about 0.61 alone and rises to about 0.75 with Reflexion. (c) Ablation: putting only the previous trajectory into the prompt (EPM) gives about 0.66, and adding reflection on top gives about 0.74.
Original Figure 4, Shinn et al., from PDF page 7, CC BY 4.0.

HotPotQA has questions that can only be answered by drawing on several Wikipedia articles. The authors tested three kinds of agents, each on 100 questions:

  • ReAct: can search with the Wikipedia API, looking things up while thinking.
  • Chain of thought: no lookups, reasoning only.
  • Chain of thought with the correct context paragraphs (CoT (GT)): the correct paragraphs that come with the dataset are handed straight to the model, to test reasoning ability on its own.

On their own, none of the three agents gets any new questions right after several retries, even when sampling at temperature 0.7. With Reflexion, ReAct gets about 20 percentage points more right. Chain of thought with the correct paragraphs starts out higher, but still gets 39% of the questions wrong, and Reflexion lifts its accuracy by another 14 percentage points.

Does the improvement come from reflection, or just from seeing the previous attempt? The ablation in the right plot separates the two. Putting only the full previous trajectory into the prompt helps somewhat; adding a first-person reflection adds another 8 percentage points. On this basis, the authors argue that just having the model rewrite isn't as effective as reflecting first and then rewriting.

Note how scoring works here. After each HotPotQA attempt, the environment checks the answer against the reference and tells the agent whether this attempt succeeded or failed. In other words, the agent always knows when it got the answer wrong; it just doesn't know what the right answer is. We'll come back to this setup later.

Programming: HumanEval, MBPP and LeetCode

Programming is where Reflexion shines brightest, because it has a way to evaluate itself that needs no reference answer, namely writing its own tests.

The authors first use chain of thought to have the model write a variety of unit tests, check them with a syntax tree and keep only the syntactically valid ones, then sample up to 6 of those to form a test suite. Once the code is written, it runs these tests; if they fail, the agent reflects and rewrites, and it only submits once everything passes. Since the whole process never touches the problem's hidden tests, the submitted solution still counts as pass@1.

Benchmark (language)Prev. bestGPT-4Reflexion
HumanEval (Python)65.880.191.0
HumanEval (Rust)-60.068.0
MBPP (Python)67.780.177.1
MBPP (Rust)-70.975.4
LeetCode Hard (Python)-7.515.0

Compiled from Table 1 of the paper; the previous best is CodeT with GPT-3.5 or Codex; values are pass@1 %.

  • The Rust problems were translated automatically from the Python problems with MultiPL-E; for HumanEval, only the 50 hardest problems were used. The point is to show that the method is language-agnostic.
  • LeetCode Hard is a new set of 40 problems the authors put together, all released after October 8, 2022 (GPT-4's pretraining cutoff).
  • The GPT-4 numbers here were measured by the authors themselves; its 80.1% on HumanEval is higher than the 67.0% in the GPT-4 technical report.

HumanEval Python rises from 80.1% to 91.0%, but MBPP Python actually drops from 80.1% to 77.1%. When the authors tracked down the cause, they found the problem was in the tests the model wrote itself.

Tests can go wrong in two ways. One is that the tests are too loose, so wrong code passes all of them and the agent submits a wrong answer early. The other is that the tests themselves are wrong, so correct code fails, and the agent goes off to reflect on code that was actually fine. The authors consider the second more acceptable, since the agent still has a chance to discover through reflection that the tests were wrong.

On MBPP Python, the rate of "all tests pass, but the code is wrong" is 16.3%; on HumanEval Python it's only 1.4%. The two have similar baseline accuracy, so the difference comes down to the quality of the tests.

What if you take parts out?

The authors ran an ablation on the 50 hardest HumanEval Rust problems, with GPT-4 as the base model:

ApproachOwn testsReflectionpass@1
Base modelNoNo60%
No testsNoYes52%
No reflectionYesNo60%
ReflexionYesYes68%

Compiled from Table 3 of the paper.

Reflecting without writing tests is actually worse than the base model. The agent can't tell whether its current code is right, so it has to use up every round, editing all the way, and ends up breaking code that was originally correct.

Tests without reflection bring no improvement at all. The tests and compiler do catch syntax and logic errors, but when the model rewrites directly, its edits don't address those errors. On this basis, the authors argue that debugging by blind trial and error, without reflection, doesn't do much on harder tasks like Rust.

The model has to be strong enough

The appendix reruns things on different models. The open-source starchat-beta scores 26% on HumanEval Python with or without Reflexion; on HotPotQA, text-davinci-003, gpt-3.5-turbo and gpt-4 all improve with Reflexion. The authors conclude that writing useful self-corrections is an ability that only stronger, larger models have.

Limitations

  • Local optima: Reflexion is essentially policy optimization done in text, and it can still get stuck at solutions that aren't good enough. The WebShop experiment in the appendix is an example. The agent has to find products on a shopping site that match a request; after 4 trials there was no improvement, and the reflections it wrote didn't help either. The authors think this kind of task needs very diverse, creative exploration, such as trying different search keywords, which Reflexion can't do.
  • Simple memory: it's just a sliding window that keeps the last few reflections; the authors suggest moving to a vector database or a SQL database in the future.
  • Limits of tests: functions that are non-deterministic, that call external APIs, whose output depends on the hardware, or that involve parallelism are all hard to write accurate tests for.

The authors also caution that automated code-writing experiments should run in an isolated environment, because the generated code isn't checked in any way before it runs.

What came after

In October 2023, in Large Language Models Cannot Self-Correct Reasoning Yet, Huang et al. from Google DeepMind and UIUC pointed out a problem shared by this kind of self-correction research. Approaches including Reflexion all use the reference answer to decide when to stop correcting, stopping once the answer is right and continuing only when it's wrong. In real situations, we don't know the correct answer.

They designed a pointed control group. It uses the reference answer in the same way to decide whether to try again, but the "correction" is just switching to a random one of the remaining options. CommonsenseQA has five choices, so every wrong answer rules one out, and after at most four rounds of guessing it's bound to be right.

GPT-3.5 and GPT-4 start at 75.8% and 82.0% on CommonsenseQA. After two rounds of random guessing, GPT-3.5 reaches 87.9%, about the same as the 89.7% from two rounds of the model correcting itself; GPT-4 reaches 91.0% by guessing, higher than its 85.5% from self-correction; after four rounds of guessing, both are at 100%. As long as someone tells you "wrong, try again," you'll keep getting more accurate even without thinking at all. Once the reference answer was taken away, they found that when models correct their own reasoning, scores go down instead of up.

This critique is aimed mainly at reasoning questions. Programming is different. Tests and an executor are checks that need no reference answer, and that's exactly how Reflexion got its 91% on HumanEval. Huang et al. also note that tasks like code generation, which can be verified with an executor and unit tests, are a different matter.

Later coding agents took the same road, writing code, running tests, reading error messages and editing, over and over. The difference is that the tests became the project's own tests, and the feedback became real error messages.

Conclusion

Reflexion leaves the weights alone and writes the experience of each failure into a reflection that goes into the prompt for the next attempt. It solves 130 of the 134 tasks in AlfWorld, gets about 20 percentage points more right on HotPotQA, and takes pass@1 on HumanEval Python from 80.1% to 91.0%.

It's most convincing on programming. The model writes its own tests and never sees a reference answer before submitting, and the ablation shows that tests and reflection both have to be there; reflection without tests actually does worse.

The reasoning results deserve a discount. The agent is told "wrong answer" every time, and that is exactly the piece of information the real world most often lacks. However good the reflection, someone first has to tell it that it was wrong.