{"notes":[{"id":"transformers/react","category":"transformers","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"ReAct","subtitle":"Knowing and Doing as One","author":"Z. Yuan","paperDate":"2022-10-06","publishedOn":"2026-07-03","world":"external/react","paperTitle":"ReAct: Synergizing Reasoning and Acting in Language Models","paperUrl":"https://arxiv.org/abs/2210.03629","bibliography":{"title":"ReAct: Synergizing Reasoning and Acting in Language Models","authors":["Shunyu Yao","Jeffrey Zhao","Dian Yu","Nan Du","Izhak Shafran","Karthik Narasimhan","Yuan Cao"],"url":"https://arxiv.org/abs/2210.03629","pdfUrl":"https://arxiv.org/pdf/2210.03629v3","codeUrl":null,"preprint":{"id":"2210.03629","submittedOn":"2022-10-06","url":"https://arxiv.org/abs/2210.03629"},"publication":null,"citation":"@misc{transformersreact2022,\n  title = {{ReAct: Synergizing Reasoning and Acting in Language Models}},\n  author = {Shunyu Yao and Jeffrey Zhao and Dian Yu and Nan Du and Izhak Shafran and Karthik Narasimhan and Yuan Cao},\n  year = {2022},\n  eprint = {2210.03629},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2210.03629}\n}\n","citationPath":"/content/papers/citations/transformers/react.0a6bd417f7a8.bib"},"thumbnail":{"src":"/content/papers/assets/ae2cd6d5c68df8887227719a0ac6c5dee8763fa902af5af73260fdc0f1ca81cc.png","width":1122,"height":489},"locale":"en","path":"/content/papers/en/transformers/react.29d81d4048f2.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"treating-thought-as-an-action","text":"Treating Thought as an Action","depth":3},{"id":"a-minimal-wikipedia-tool","text":"A Minimal Wikipedia Tool","depth":3},{"id":"how-do-you-get-the-model-to-do-this","text":"How Do You Get the Model to Do This?","depth":3},{"id":"when-should-it-trust-itself","text":"When Should It Trust Itself?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"results-on-knowledge-tasks","text":"Results on Knowledge Tasks","depth":3},{"id":"what-goes-wrong","text":"What Goes Wrong?","depth":3},{"id":"smaller-models-come-out-ahead-after-fine-tuning","text":"Smaller Models Come Out Ahead After Fine-Tuning","depth":3},{"id":"beyond-question-answering","text":"Beyond Question Answering","depth":3},{"id":"you-can-build-it-without-a-framework","text":"You Can Build It Without a Framework","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"In 2022, Chain-of-Thought (CoT) showed everyone something useful: ask a large language model to “think step by step,” and its arithmetic and commonsense reasoning improve noticeably.\n\nBut CoT has a problem: all that thinking happens inside its own head.\n\nThe model relies on knowledge stored in its parameters. If it misremembers a fact halfway through, every subsequent step builds on that mistake, eventually producing a very orderly argument for the wrong answer.\n\nMeanwhile, other research let language models **act** in an environment: plan robot movements, click buttons on a web page, or call a search engine. Most of those approaches predicted actions without giving the model much room to pause and think about the situation.\n\nThis paper from Princeton University and Google Research's Brain team, published at ICLR 2023, proposes alternating the two: **think a step, take an action, observe the result, then think about the next step**. The authors call it ReAct, combining Reasoning and Acting.\n\nIt is also one of the starting points most often cited in the history of agents that look things up for themselves.\n\n## Defining the Problem\n\nStart with the paper's motivating example. The question comes from HotpotQA, a multi-hop question-answering dataset that requires combining information from multiple Wikipedia sources:\n\n> Aside from the Apple Remote, what other device can control the program Apple Remote was originally designed to work with?\n\nTo answer, you first need to find the program Apple Remote was designed for, then look up what else can control it.\n\nThe authors have the model answer in four ways:\n\n[![Four approaches to a HotpotQA question: Standard, CoT, and Act-Only answer incorrectly; ReAct alternates reasoning and search to answer correctly](/content/papers/assets/ae2cd6d5c68df8887227719a0ac6c5dee8763fa902af5af73260fdc0f1ca81cc.png)](/content/papers/assets/ae2cd6d5c68df8887227719a0ac6c5dee8763fa902af5af73260fdc0f1ca81cc.png)\n\n*Figure 1(1) from the paper, excerpted from [**page 2 of the v3 PDF**](https://arxiv.org/pdf/2210.03629v3#page=2), CC BY 4.0. Red marks errors or hallucinations; green marks correct information obtained through search. The ALFWorld example in the lower half of the original figure is omitted.*\n\n**(1a) Standard**: It answers “iPod” directly. Wrong.\n\n**(1b) CoT**: The model starts reasoning step by step: “Apple Remote was originally designed for Apple TV; Apple TV can be controlled with an iPhone, iPad, and iPod Touch.” The reasoning flows smoothly, but the first statement is wrong. The red portions are hallucinations. Naturally, the answer is wrong too.\n\n**(1c) Act-Only**: Search without reasoning. The model searches for Apple Remote, then Front Row, and even finds “Front Row (software).” But it cannot turn these observations into an answer. It eventually answers “yes.”\n\n**(1d) ReAct**:\n\n- **Thought 1**: I need to search for Apple Remote and find out which program it was originally designed for.\n- **Act 1**: `Search[Apple Remote]`\n- **Obs 1**: Apple Remote is a remote control introduced by Apple in 2005… originally designed to control the **Front Row media center** program.\n- **Thought 2**: It was designed for Front Row. Next, search for Front Row to find what else can control it.\n- **Act 2**: `Search[Front Row]`\n- **Obs 2**: [Front Row] was not found. Similar entries: “Front Row Seat to Earth,” “Front Row Motorsports,” “Front Row (software)”…\n- **Thought 3**: Front Row was not found. I'll search for Front Row (software) instead.\n- **Act 3**: `Search[Front Row (software)]`\n- **Obs 3**: Front Row is a discontinued media center application…\n- **Thought 4**: Front Row (software) can be controlled with Apple Remote or **keyboard function keys**. So the answer is keyboard function keys.\n- **Act 4**: `Finish[keyboard function keys]`\n\nCorrect.\n\nComparing the four approaches makes the problem clear:\n\n- **Thinking without checking** (CoT) can lead the model astray through misremembered facts.\n- **Checking without thinking** (Act) can retrieve information without knowing how to use it.\n\nThe authors want to know: **can interleaving reasoning and action make up for the weaknesses of each?**\n\n## Solving the Problem\n\n### Treating Thought as an Action\n\nThe authors first describe the problem in an agent framework.\n\nAt time $t$, the agent receives an observation $o_t$ from the environment and chooses an action $a_t$ based on the context so far, $c_t=(o_1,a_1,\\dots,o_{t-1},a_{t-1},o_t)$.\n\nReAct's idea fits into one sentence: **expand the action space to include language.**\n\n$$\n\\hat{\\mathcal{A}}=\\mathcal{A}\\cup\\mathcal{L}\n$$\n\n$\\mathcal{A}$ contains the original actions that affect the environment, such as search; $\\mathcal{L}$ is the language space. An “action” selected from $\\mathcal{L}$ is a **thought**.\n\nA thought does not change the environment, so it produces no new observation. Its role is to organize the current context and write the result back into it for the next step:\n\n$$\nc_{t+1}=(c_t,\\hat{a}_t)\n$$\n\nIn the example above, thoughts serve several purposes: breaking down the problem (Thought 1), extracting key facts from observations (Thoughts 2 and 4), adjusting the plan after an exception (Thought 3), and assembling the final answer (Thought 4).\n\n### A Minimal Wikipedia Tool\n\nFor the two knowledge tasks, HotpotQA and FEVER, the authors design a very basic Wikipedia API with just three actions:\n\n1. **`search[entity]`**: If the entry exists, return the first five sentences of the page. Otherwise, list the top five similar entries found by the search engine.\n2. **`lookup[string]`**: Return the next sentence on the current page containing that string, much like Ctrl+F in a browser.\n3. **`finish[answer]`**: Submit the answer and end the task.\n\nThe authors emphasize that these actions generally retrieve short passages using exact entity names, making them much weaker than the best lexical or neural retrievers of the time. This is deliberate: they want to imitate how people browse Wikipedia and make the model use language reasoning to decide what to look up.\n\nFEVER is a fact-verification dataset in which each claim is classified as “supports,” “refutes,” or “not enough information.” For both tasks, the model receives only the question or claim, without any reference passages.\n\n### How Do You Get the Model to Do This?\n\nThe main experiments train no new model. They use a frozen PaLM-540B with only a few demonstrations in the prompt.\n\nThe authors randomly choose six HotpotQA questions and three FEVER claims from the training sets, manually write complete thought–action–observation trajectories, and include them as examples. They also report that adding more examples did not help.\n\nThe inference process can be written as a loop:\n\n```python\ndef react(question, llm, tools, examples, max_steps=7):\n    prompt = examples + f\"Question: {question}\\n\"\n    for step in range(1, max_steps + 1):\n        # The model writes this step's thought and action, for example:\n        # \"Thought 2: ...\\nAct 2: Search[Front Row]\"\n        output = llm(prompt, stop=[f\"Obs {step}:\"])\n        prompt += output\n\n        action, arg = parse_action(output)          # For example (\"search\", \"Front Row\")\n        if action == \"finish\":\n            return arg\n\n        observation = tools[action](arg)            # Call the Wikipedia API\n        prompt += f\"\\nObs {step}: {observation}\\n\"  # Append the observation to the prompt\n    return None                                     # No answer within the step budget\n```\n\nThis is an illustrative implementation based on the paper's description. The key is `stop`: after writing its thought and action, the model stops. The program actually performs the search and appends the result as an observation. Observations come from the environment; the model does not invent them.\n\n\n> [!TIP]\n> \n> **How Often Should It Think?**\n> \n> For reasoning-heavy tasks such as HotpotQA and FEVER, the authors strictly alternate thoughts and actions: think first, then act at every step.\n> \n> But a text adventure task in ALFWorld may take dozens of actions, such as “go to countertop 1” or “open drawer 2.” Thinking before every action is impractical. The authors therefore let the model decide when to insert a thought, usually when planning, tracking progress, or changing the plan.\n\n\n### When Should It Trust Itself?\n\nWhile analyzing the results, the authors find that ReAct and CoT have different strengths: ReAct is more grounded in facts, while CoT is better at organizing a reasoning structure.\n\nThey therefore design two hybrids. Here CoT-SC is the self-consistency version of CoT: sample 21 reasoning paths at temperature 0.7 and take the majority answer.\n\n- **ReAct → CoT-SC**: If ReAct cannot answer within the step budget, fall back to CoT-SC. HotpotQA allows seven steps and FEVER five. Among successful trajectories, only 0.84% and 1.33%, respectively, use all seven or five steps.\n- **CoT-SC → ReAct**: If the most common answer among $n$ CoT-SC samples appears fewer than $n/2$ times, the model lacks confidence in its knowledge and switches to ReAct to check.\n\n## Discussion\n\n### Results on Knowledge Tasks\n\n| Method | HotpotQA (EM) | FEVER (accuracy) |\n| --- | ---: | ---: |\n| Standard | 28.7 | 57.1 |\n| CoT | 29.4 | 56.3 |\n| CoT-SC | 33.4 | 60.4 |\n| Act | 25.7 | 58.9 |\n| ReAct | 27.4 | 60.9 |\n| CoT-SC → ReAct | 34.2 | **64.6** |\n| ReAct → CoT-SC | **35.1** | 62.0 |\n| Supervised SoTA | 67.5 | 89.5 |\n\n*Compiled from Table 1 of the paper. The model is PaLM-540B.*\n\nFirst compare ReAct with Act: ReAct is better on both tasks. Both can search, but adding thought changes their ability to assemble an answer.\n\nNow compare ReAct with CoT: ReAct wins on FEVER (60.9 versus 56.3), but does slightly worse on HotpotQA (27.4 versus 29.4). The authors explain that “supports” and “refutes” in FEVER can differ by a small detail, making accurate information especially important.\n\nThe best results come from the hybrids: ReAct → CoT-SC reaches 35.1 on HotpotQA, while CoT-SC → ReAct reaches 64.6 on FEVER. In additional curves over sample counts, both hybrids match the performance of CoT-SC with 21 samples using just three to five.\n\nBut look at the last row too: all these prompting methods remain far behind the best supervised models. HotpotQA's 67.5 and FEVER's 89.5 come from systems trained specifically for those tasks.\n\n### What Goes Wrong?\n\nTo understand the difference between ReAct and CoT on HotpotQA, the authors sample 50 correct and 50 incorrect trajectories from each, 200 in total, and manually label their success and failure types:\n\n| Type | Description | ReAct | CoT |\n| --- | --- | ---: | ---: |\n| Success: true positive | Correct reasoning and facts | 94% | 86% |\n| Success: false positive | Correct answer, but fabricated reasoning or facts | 6% | 14% |\n| Failure: reasoning error | Faulty reasoning, including getting stuck repeating steps | 47% | 16% |\n| Failure: search result error | Search returns nothing or no useful information | 23% | — |\n| Failure: hallucination | Fabricated reasoning or facts | 0% | 56% |\n| Failure: label ambiguity | Correct answer with wording different from the reference | 29% | 28% |\n\n*Compiled from Table 2 of the paper. Success and failure percentages are calculated separately.*\n\nThis table deserves a close look.\n\n**Hallucination is CoT's biggest problem.** Among CoT's incorrect cases, 56% involve hallucination. Even among correct answers, 14% happen to arrive at the right answer through invented reasoning. Among ReAct's incorrect cases, hallucination accounts for 0%.\n\n**But ReAct has problems of its own.** Reasoning errors account for 47% of its failures, far above CoT's 16%. The authors observe a failure pattern specific to ReAct: the model keeps repeating earlier thoughts and actions and cannot escape the loop. Interleaving thoughts, actions, and observations grounds the reasoning, but also makes it less flexible.\n\n**Failed searches can derail ReAct.** In 23% of failures, search finds no useful information, and the model struggles to reorganize its reasoning.\n\nThe authors also find that some HotpotQA reference answers are outdated.\n\n### Smaller Models Come Out Ahead After Fine-Tuning\n\nWith prompting alone, ReAct is the worst of the four approaches on PaLM-8B and 62B: smaller models struggle to learn both reasoning and action from a handful of examples.\n\nThe authors then fine-tune smaller models on 3,000 successful trajectories generated by ReAct itself. The results reverse:\n\n- Fine-tuned PaLM-8B ReAct beats every prompting method using PaLM-62B.\n- Fine-tuned PaLM-62B ReAct beats every prompting method using PaLM-540B.\n\nFine-tuning Standard or CoT is much less effective. The authors explain the distinction this way: those approaches teach the model to memorize potentially hallucinated facts, whereas ReAct teaches it how to reason and look things up on Wikipedia—skills that generalize better.\n\n### Beyond Question Answering\n\nThe authors also test two interactive decision-making tasks:\n\n- **ALFWorld**: A text-based household game, with tasks such as finding a pepper shaker and putting it in a drawer. The best ReAct trial achieves an average success rate of 71%, versus a best of 45% for action-only Act and 37% for the imitation-learning method BUTLER.\n- **WebShop**: A simulated shopping website with 1.18 million real products, where the agent finds products matching a user's description.\n\nThe abstract summarizes the result: with only one or two demonstrations, ReAct improves success rates on the two tasks by 34 and 10 percentage points over imitation-learning and reinforcement-learning methods, respectively. Those comparison methods train on $10^3$ to $10^5$ task examples.\n\n### You Can Build It Without a Framework\n\nAnother attraction of ReAct is its simplicity. You barely need a framework.\n\nIn March 2023, Simon Willison wrote a [**simple Python implementation**](https://til.simonwillison.net/llms/python-react-pattern) using `gpt-3.5-turbo`. A system prompt asks the model to output Thought, Action, PAUSE, and Observation in sequence, followed by an Answer. The program uses a regular expression to extract the `Action:` line, runs the corresponding tool—Wikipedia search, his own blog search, or a calculator—and sends the result back as an `Observation:`, for at most five rounds.\n\nHe also warns that the calculator directly uses Python's `eval()`, which is extremely dangerous and should run in a sandbox such as WebAssembly. The implementation is also still quite fragile.\n\nOnce the model decides which tools to call, safety and robustness become engineering concerns. The paper spends relatively little time on them.\n\n## Conclusion\n\nReAct makes one central change: **add thought to the agent's action space**, allowing the language model to alternate reasoning and action.\n\nThe paper's evidence shows:\n\n1. **Looking things up without thinking wastes information.** With the same search access, ReAct beats action-only Act on both HotpotQA and FEVER.\n2. **Thinking without checking invites hallucination.** Hallucination accounts for 56% of CoT's incorrect cases and 0% of ReAct's, although ReAct makes more reasoning errors and can get stuck in loops.\n3. **Combining the two works best.** Switching between ReAct and CoT-SC based on confidence or step limits produces the best prompting results on both knowledge tasks.\n4. **Fine-tuning amplifies the effect.** After training on 3,000 trajectories, smaller ReAct models beat the prompting approaches of larger models.\n\nThese prompting methods still lag far behind task-specific trained systems, and the Wikipedia tools are deliberately weak.\n\nBut the loop—think, look something up, observe, think again—stuck around. Later coding agents that search files and run commands follow much the same pattern. The tools have simply changed from three Wikipedia actions to grep, file reading, and a terminal.\n","imageSizes":{"/content/papers/assets/ae2cd6d5c68df8887227719a0ac6c5dee8763fa902af5af73260fdc0f1ca81cc.png":{"width":1122,"height":489}}}