Toolformer: Language Models Can Teach Themselves to Use Tools
- Publication
- NeurIPS 2023Conference
- First submitted to arXiv
- arXiv:2302.04761
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{modeltuningtoolformer2023,
title = {{Toolformer: Language Models Can Teach Themselves to Use Tools}},
author = {Timo Schick and Jane Dwivedi-Yu and Roberto Dessì and Roberta Raileanu and Maria Lomeli and Luke Zettlemoyer and Nicola Cancedda and Thomas Scialom},
year = {2023},
eprint = {2302.04761},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2302.04761}
}
There's something contradictory about large language models. They can take on a new task from a few examples, and they do a respectable job of writing, translating and answering questions, yet they often get simple division wrong, misremember where someone was born, and don't know what today's date is. A calculator, a search engine and a calendar can handle all of these.
By early 2023, people were already getting models to use tools. The approaches mostly came in two kinds. One needs a lot of human annotation; LaMDA, for example, had annotators demonstrate when to look things up. The other writes few-shot prompts for specific tasks, telling the model in advance that this problem calls for a calculator and that one for search. The first is expensive, and the second has to be rewritten for every new task.
Schick et al. at Meta wanted the model to learn on its own. Give it just a few demonstrations per tool, let it insert tool calls into a large amount of ordinary text by itself, and then let the model itself judge which calls actually helped.
Defining the problem
The authors want the tool use the model learns to meet two conditions:
- Self-supervised: no large-scale human annotation. That's not only because annotation is expensive, but also because the information people find useful isn't necessarily what the model needs.
- Keeping generality: the model decides for itself whether to use a tool, which one and how, instead of being tied to specific tasks.
So that tool calls can be written straight into text, the authors write each API call as a piece of text. A call consists of an API name and an input , and the result it returns is :
In practice, existing characters, "[", "]" and "->", stand in for the special tokens, so the model's vocabulary doesn't need to change.

Solving the problem

The whole pipeline works by doctoring an ordinary language-model training set. The original dataset is just plain text. The authors have the model insert tool calls into it, which gives a new dataset , and then fine-tune the model itself on that.
1. Sampling calls
Each tool gets its own prompt with a few demonstrations. Here's the prompt for the question-answering tool:
Your task is to add calls to a Question Answering API to a piece of text.
The questions should help you get information required to complete the text.
You can call the API by writing "[QA(question)]" where "question" is the
question you want to ask. Here are some examples of API calls:
Input: Joe Biden was born in Scranton, Pennsylvania.
Output: Joe Biden was born in [QA("Where was Joe Biden born?")] Scranton,
[QA("In which state is Scranton?")] Pennsylvania.
Input: Coca-Cola, or Coke, is a carbonated soft drink manufactured by
the Coca-Cola Company.
Output: Coca-Cola, or [QA("What other name is Coca-Cola known by?")] Coke,
is a carbonated soft drink manufactured by [QA("Who manufactures Coca-Cola?")]
the Coca-Cola Company.
Input: x
Output:
The prompt from Figure 3 of the paper.
Put a piece of text into the prompt, and for each position , compute the probability that the model starts writing <API> there. Positions whose probability exceeds the threshold are kept, up to of them; for each position, up to call contents are then sampled.
2. Executing calls
The authors set up five tools, all with text input and text output:
| Tool | Example input | Example output |
|---|---|---|
| Question answering | Where was the Knights of Columbus founded? | New Haven, Connecticut |
| Wikipedia search | Fishing Reel Types | A Wikipedia excerpt on types of fishing rods and reels |
| Calculator | 27 + 4 * 2 | 35 |
| Calendar | (no input) | Today is Monday, January 30, 2023. |
| Translation | sûreté nucléaire | nuclear safety |
Compiled from Table 1 of the paper.
- Question answering: Atlas, a retrieve-then-answer language model fine-tuned on Natural Questions.
- Wikipedia search: BM25 retrieval over KILT's Wikipedia data, returning short text snippets.
- Calculator: the four basic arithmetic operations only, with results rounded to two decimal places.
- Calendar: takes no input and returns today's date.
- Translation: NLLB with 600M parameters, using fastText to detect the source language automatically, and always translating into English.
3. Filtering: did this call help?
This is the most important step in the whole paper. The authors use the model's own loss to judge whether a call is useful.
Let be the weighted cross-entropy of the model predicting each token from position onward, with added in front:
Then compare two cases:
is making the call and getting the result; is not calling at all, or calling without getting the result, whichever is better. Only calls with are kept.
Comparing against "calling without getting the result" rules out one case. Sometimes just writing out the question already hints at what comes next; the credit for that kind of call doesn't belong to the tool, so it shouldn't be kept.
The weights come from , then normalized. The raw weights for the 1st through 5th tokens after the call are 1, 0.8, 0.6, 0.4 and 0.2, and 0 after that. Closer tokens weigh more, so that calls show up where the information actually gets used. Take the sentence "Pittsburgh is also known as the Steel City." Insert a question-answering call before "the," and once it gets back "Steel City," the model is all but certain the next word is Steel; the loss drops sharply, and the call is kept. Asking which country Pittsburgh is in and getting "United States" does nothing to help predict "the Steel City," so that call is thrown away.
There's one more detail in the filtering. At this point the model has never seen a tool call, and forcing one into the middle of a sentence breaks the flow of the text and actually pushes perplexity up. So when computing , the authors put the call and its result at the very beginning of the text, as a prefix.
4. Fine-tuning
The calls kept for all the tools are merged and inserted back into the original text, giving . Apart from the added calls, the text of is exactly the same as , so when the model is fine-tuned on it, it sees the same knowledge as before, and its language ability shouldn't suffer.
The text is a subset of CCNet, and the model is GPT-J 6B. To save compute on annotation, the authors first use rules to pick out text that is more likely to need tools; for the calculator, for example, they only process passages containing at least three numbers. Fine-tuning uses a batch size of 128 and a learning rate of , with linear warmup over the first 10% of steps.
How many calls survive varies by tool. At a filtering threshold of , Wikipedia search has 60,974, the calendar 20,587, question answering 18,526, translation 1,034, and the calculator only 994.
Inference
After fine-tuning, decoding works as usual. When the model writes "→", it means it's waiting for a tool result. Decoding pauses at that point, the tool actually gets called, the result and "]" are appended, and the model carries on writing. For example, decoding pauses when the model has written "[Calculator(400 / 1400) →"; the calculator returns 0.29, "0.29]" is appended, and the model goes on to write "29%".
Decoding also gets two tweaks. The model is allowed to start a call whenever <API> is among the 10 most likely tokens, not only when it's ranked first, which encourages it to use tools more. And each input gets at most one call, so the model can't get stuck in a loop of calling over and over without ever writing an answer.
Discussion
Experimental setup
The main comparison is among four models: the original GPT-J; GPT-J + CC, fine-tuned on the same CCNet text but without tool calls; Toolformer; and Toolformer with the probability of <API> set to 0, so it can't use tools. There are also comparisons with OPT 66B and GPT-3 175B (davinci, without instruction tuning).
Everything is zero-shot. The task is described in words only, no examples are given, and the model isn't told which tool to use. Scoring is fairly lenient; for example, an answer counts as correct if it appears within the first few words of the model's output.
Results
| Task | GPT-J | Tools off | Toolformer | GPT-3 175B |
|---|---|---|---|---|
| LAMA SQuAD | 17.8 | 22.1 | 33.8 | 26.8 |
| LAMA Google-RE | 4.9 | 6.3 | 11.5 | 7.0 |
| LAMA T-REx | 31.9 | 34.9 | 53.5 | 39.8 |
| ASDiv | 7.5 | 14.8 | 40.4 | 14.0 |
| SVAMP | 5.2 | 6.3 | 29.4 | 10.0 |
| MAWPS | 9.9 | 15.0 | 44.0 | 19.8 |
| WebQS | 18.5 | 18.9 | 26.3 | 29.0 |
| Natural Questions | 12.8 | 12.6 | 17.7 | 22.6 |
| TriviaQA | 43.9 | 46.7 | 48.8 | 65.9 |
| TempLAMA | 13.7 | 12.7 | 16.3 | 15.5 |
| Dateset | 3.9 | 5.9 | 27.3 | 0.8 |
Compiled from Tables 3, 4, 5 and 7 of the paper; "Tools off" is Toolformer with API calls disabled; values are in %.
Fill-in-the-blank knowledge questions (LAMA): on 98.1% of questions the model goes to the question-answering tool on its own, and it beats GPT-3 on all three subsets. LAMA's sentences come from Wikipedia, so Wikipedia search is disabled here to keep things from being too easy.
Math word problems: the calculator is called on 97.9% of problems, and scores are more than double those with tools off, far above GPT-3 as well. Interestingly, Toolformer with tools off also does better than GPT-J. The authors guess that after seeing lots of calculations and their results, the model's own arithmetic improved too.
Open-domain QA: the question-answering tool was fine-tuned on Natural Questions, so using it would amount to cheating, and it's disabled here. The model turns to Wikipedia search 99.3% of the time instead, beating the other GPT-J models but still losing to GPT-3. The authors see two reasons. The BM25 search engine is very basic and often returns irrelevant results, and Toolformer can't rephrase its query and search again, or look further down the results.
Time-related questions: Dateset can only be answered if you know today's date, as in "What day of the week was it 30 days ago?" The calendar is used on 54.8% of its questions, and the score rises from 5.9% to 27.3%. TempLAMA asks about facts that change over time, such as which team a certain player plays for. Toolformer improves there too, but only 0.2% of questions use the calendar; the gains come mostly from search and question answering. The best approach would be to look up the date first and then ask with that date, but only one call is allowed per question, and the calls in the training data were all sampled separately for each tool, so the model has never seen this kind of relay.
Multilingual QA (MLQA): the questions are in Arabic, German, Spanish, Hindi, Vietnamese or Simplified Chinese, and the context paragraphs are in English. Depending on the language, the model first translates the question into English on 63.8% to 94.9% of questions, and does better than with tools off; Hindi is the exception, with translation used only 7.3% of the time. Toolformer doesn't consistently beat the original GPT-J, though, because fine-tuning on CCNet by itself made some languages worse.
Language ability doesn't get worse
| Model | WikiText | CCNet |
|---|---|---|
| GPT-J | 9.9 | 10.6 |
| GPT-J + CC | 10.3 | 10.5 |
| Toolformer (tools off) | 10.3 | 10.5 |
Compiled from Table 8 of the paper; the numbers are perplexity, lower is better.
Fine-tuning on CCNet makes WikiText perplexity slightly worse, but adding tool calls or not makes no difference. Perplexity with tools switched on can't be computed, because any position might call any tool, and getting the probability would mean summing over every possibility.
How big does the model need to be?

The authors redid the experiment with four sizes of GPT-2, giving them only three tools: question answering, the calculator and Wikipedia search. The ability to use tools only appears at around 775M parameters; smaller models do about the same with or without tools. Wikipedia search is the exception, which even small models can make use of; the authors guess that's because this API is simpler. As models grow, they get better without tools and better at using tools at the same time, so the gap between the two never goes away.
The decoding threshold
| T-REx | Call rate | WebQS | Call rate | |
|---|---|---|---|---|
| 0 | 34.9 | 0.0% | 18.9 | 0.0% |
| 1 | 47.8 | 40.3% | 19.3 | 8.5% |
| 3 | 52.9 | 82.8% | 26.3 | 99.3% |
| 10 | 53.5 | 98.1% | 26.3 | 100% |
Compiled from Table 9 of the paper, showing selected columns; the T-REx and WebQS columns are accuracy (%).
is how high <API> has to rank before a call starts. amounts to turning tools off, and is ordinary greedy decoding. With ordinary greedy decoding, the model calls a tool on only 8.5% of WebQS questions, and the score barely changes; only when the threshold is relaxed to the top 3 does it start using tools heavily.
At , on the questions where the model decides not to call, its scores (44.3% on T-REx, 19.9% on WebQS) are actually higher than the average with tools forbidden entirely (34.9% and 18.9%). That shows the model knows, to some degree, which questions it can't handle, and saves the tools for those. Once the threshold is relaxed, this judgment disappears.
What the kept calls look like
The authors list some examples sorted by filter score , from high to low. The high-scoring ones mostly make sense. One searches for Flodden when the text mentions the UK's oldest war memorial window, with a score of 5.49; another asks how long the Nile is when the text reaches "The Nile has an approximate length of", with a score of 2.08.
The low-scoring ones are mostly useless, such as computing 85 / 23 after "85 patients (23%)", with a score of −0.02. But "Catch this fast train to success!" triggers a search for "Fast train success", which turns up a 1971 song with completely unrelated content, yet it still lowers perplexity, with a score of 0.92. The authors think a small amount of this kind of noise isn't necessarily a bad thing, since it keeps the model from blindly trusting every tool result.
Limitations
The authors list these themselves:
- No chaining tools: for example, looking up the date first and then asking with it. The calls for each tool are sampled separately, so the training data has no such examples.
- No interaction: a search engine may return hundreds of results, and the model can neither browse them nor revise its query.
- Sensitive to wording: whether to call a tool is easily swayed by how the input is phrased.
- Very inefficient: processing more than a million documents yields only a few thousand useful calculator calls. The authors suggest iterating, as other bootstrapping methods do.
- Ignores cost: different tools have very different computational costs, and the model doesn't take that into account when deciding.
What came after
Four months after Toolformer came out, tool use became a standard feature of commercial models, though by a different route. On June 13, 2023, OpenAI added function calling to its API. Developers describe their functions in JSON; when the model decides it needs one, it outputs a JSON object with the arguments, the application runs it, and the result is handed back to the model. The tools are defined by developers, and the model learns during training to call them from their descriptions, with no need to figure it out from ordinary text on its own.
Chaining and interaction, which Toolformer listed as limitations, became basic capabilities in the agent loop. The model calls a tool, looks at the result and decides the next step, going back and forth dozens of times on a single task. In November 2024, Anthropic introduced the Model Context Protocol as an open standard for connecting AI assistants to all kinds of data sources and tools, and the way tools get hooked up started to converge.
Conclusion
Toolformer gives each tool just a few demonstrations, lets GPT-J insert tool calls into ordinary text on its own, and filters them by "does this make the following text easier to predict?" After fine-tuning, this 6B model beats GPT-3 175B on knowledge fill-in-the-blank, math word problems and date questions, and its language ability doesn't get worse.
The most interesting design choice in this paper is letting the model judge for itself whether a tool helps. Where to insert a question and which answer helps are all decided by the model's loss, with nobody telling it; even the call that turned up that 1971 song was kept because the model itself found it helpful.
Later tool use didn't keep this self-supervision. Instead, developers write the tool descriptions and the model calls tools accordingly, which is simple and controllable. But the format of "write the call into the text, splice the result back into the text, and let the model keep writing" did stick, and every step of today's agents works this way.