Back to paper library
arXiv 2023NeurIPS 2023Model adaptationPaper notes

Toolformer

When in doubt, look it up

Original paper

Toolformer: Language Models Can Teach Themselves to Use Tools

Paper authors

Timo Schick · Jane Dwivedi-Yu · Roberto Dessì …

Show all 8 authors

Timo Schick · Jane Dwivedi-Yu · Roberto Dessì · Roberta Raileanu · Maria Lomeli · Luke Zettlemoyer · Nicola Cancedda · Thomas Scialom

Publication
NeurIPS 2023Conference
First submitted to arXiv
arXiv:2302.04761
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{modeltuningtoolformer2023,
  title = {{Toolformer: Language Models Can Teach Themselves to Use Tools}},
  author = {Timo Schick and Jane Dwivedi-Yu and Roberto Dessì and Roberta Raileanu and Maria Lomeli and Luke Zettlemoyer and Nicola Cancedda and Thomas Scialom},
  year = {2023},
  eprint = {2302.04761},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.04761}
}
On this page

There's something contradictory about large language models. They can take on a new task from a few examples, and they do a respectable job of writing, translating and answering questions, yet they often get simple division wrong, misremember where someone was born, and don't know what today's date is. A calculator, a search engine and a calendar can handle all of these.

By early 2023, people were already getting models to use tools. The approaches mostly came in two kinds. One needs a lot of human annotation; LaMDA, for example, had annotators demonstrate when to look things up. The other writes few-shot prompts for specific tasks, telling the model in advance that this problem calls for a calculator and that one for search. The first is expensive, and the second has to be rewritten for every new task.

Schick et al. at Meta wanted the model to learn on its own. Give it just a few demonstrations per tool, let it insert tool calls into a large amount of ordinary text by itself, and then let the model itself judge which calls actually helped.

Defining the problem

The authors want the tool use the model learns to meet two conditions:

  • Self-supervised: no large-scale human annotation. That's not only because annotation is expensive, but also because the information people find useful isn't necessarily what the model needs.
  • Keeping generality: the model decides for itself whether to use a tool, which one and how, instead of being tied to specific tasks.

So that tool calls can be written straight into text, the authors write each API call as a piece of text. A call cc consists of an API name aca_c and an input ici_c, and the result it returns is rr:

e(c)=<API> ac(ic) </API>,e(c,r)=<API> ac(ic)→r </API>e(c) = \texttt{<API>}\, a_c(i_c)\, \texttt{</API>}, \qquad e(c, r) = \texttt{<API>}\, a_c(i_c) \rightarrow r\, \texttt{</API>}

In practice, existing characters, "[", "]" and "->", stand in for the special tokens, so the model's vocabulary doesn't need to change.

Original Figure 1. Example outputs from Toolformer, where the model decides on its own to call a tool partway through the text. The first example calls a question-answering system before "the MMS", asking who publishes The New England Journal of Medicine, and gets Massachusetts Medical Society. The second calls a calculator before "29%", Calculator(400 / 1400) → 0.29. The third calls a translation system before "turtle", MT("tortuga") → turtle. The fourth calls Wikipedia search for the Brown Act, gets a description of this California law, and then goes on to write that it requires local legislative bodies to hold their meetings in public.
Original Figure 1, Schick et al., from PDF page 1, arXiv license, scholarly quotation.

Solving the problem

Original Figure 2. The three steps of Toolformer, using the question-answering tool as an example. The input text is "Pittsburgh is also known as the Steel City". Step 1 samples candidate calls after "Pittsburgh is also known as", such as "What other name is Pittsburgh known by?" and "Which country is Pittsburgh in?". Step 2 executes the calls, which return Steel City and United States respectively. Step 3 filters: the first call brings the loss on the following text below both not calling and calling without a result, so it is kept; the second doesn't, so it is dropped. Finally, the kept calls are inserted back into the original text, giving new data with API calls.
Original Figure 2, Schick et al., from PDF page 2, arXiv license, scholarly quotation.

The whole pipeline works by doctoring an ordinary language-model training set. The original dataset C\mathcal{C} is just plain text. The authors have the model insert tool calls into it, which gives a new dataset C∗\mathcal{C}^*, and then fine-tune the model itself on that.

1. Sampling calls

Each tool gets its own prompt with a few demonstrations. Here's the prompt for the question-answering tool:

text
Your task is to add calls to a Question Answering API to a piece of text.
The questions should help you get information required to complete the text.
You can call the API by writing "[QA(question)]" where "question" is the
question you want to ask. Here are some examples of API calls:
Input: Joe Biden was born in Scranton, Pennsylvania.
Output: Joe Biden was born in [QA("Where was Joe Biden born?")] Scranton,
[QA("In which state is Scranton?")] Pennsylvania.
Input: Coca-Cola, or Coke, is a carbonated soft drink manufactured by
the Coca-Cola Company.
Output: Coca-Cola, or [QA("What other name is Coca-Cola known by?")] Coke,
is a carbonated soft drink manufactured by [QA("Who manufactures Coca-Cola?")]
the Coca-Cola Company.
Input: x
Output:

The prompt from Figure 3 of the paper.

Put a piece of text xx into the prompt, and for each position ii, compute the probability pip_i that the model starts writing <API> there. Positions whose probability exceeds the threshold τs\tau_s are kept, up to kk of them; for each position, up to mm call contents are then sampled.

2. Executing calls

The authors set up five tools, all with text input and text output:

ToolExample inputExample output
Question answeringWhere was the Knights of Columbus founded?New Haven, Connecticut
Wikipedia searchFishing Reel TypesA Wikipedia excerpt on types of fishing rods and reels
Calculator27 + 4 * 235
Calendar(no input)Today is Monday, January 30, 2023.
Translationsûreté nucléairenuclear safety

Compiled from Table 1 of the paper.

  • Question answering: Atlas, a retrieve-then-answer language model fine-tuned on Natural Questions.
  • Wikipedia search: BM25 retrieval over KILT's Wikipedia data, returning short text snippets.
  • Calculator: the four basic arithmetic operations only, with results rounded to two decimal places.
  • Calendar: takes no input and returns today's date.
  • Translation: NLLB with 600M parameters, using fastText to detect the source language automatically, and always translating into English.

3. Filtering: did this call help?

This is the most important step in the whole paper. The authors use the model's own loss to judge whether a call is useful.

Let Li(z)L_i(z) be the weighted cross-entropy of the model predicting each token from position ii onward, with zz added in front:

Li(z)=−∑j=inwj−i⋅log⁡pM(xj∣z,x1:j−1)L_i(z) = -\sum_{j=i}^{n} w_{j-i} \cdot \log p_M(x_j \mid z, x_{1:j-1})

Then compare two cases:

Li+=Li(e(ci,ri)),Li−=min⁡(Li(ε), Li(e(ci,ε)))L_i^+ = L_i(e(c_i, r_i)), \qquad L_i^- = \min\big(L_i(\varepsilon),\ L_i(e(c_i, \varepsilon))\big)

Li+L_i^+ is making the call and getting the result; Li−L_i^- is not calling at all, or calling without getting the result, whichever is better. Only calls with Li−−Li+≥τfL_i^- - L_i^+ \ge \tau_f are kept.

Comparing against "calling without getting the result" rules out one case. Sometimes just writing out the question already hints at what comes next; the credit for that kind of call doesn't belong to the tool, so it shouldn't be kept.

The weights ww come from w~t=max⁡(0,1−0.2t)\tilde w_t = \max(0, 1 - 0.2t), then normalized. The raw weights for the 1st through 5th tokens after the call are 1, 0.8, 0.6, 0.4 and 0.2, and 0 after that. Closer tokens weigh more, so that calls show up where the information actually gets used. Take the sentence "Pittsburgh is also known as the Steel City." Insert a question-answering call before "the," and once it gets back "Steel City," the model is all but certain the next word is Steel; the loss drops sharply, and the call is kept. Asking which country Pittsburgh is in and getting "United States" does nothing to help predict "the Steel City," so that call is thrown away.

There's one more detail in the filtering. At this point the model has never seen a tool call, and forcing one into the middle of a sentence breaks the flow of the text and actually pushes perplexity up. So when computing LiL_i, the authors put the call and its result at the very beginning of the text, as a prefix.

4. Fine-tuning

The calls kept for all the tools are merged and inserted back into the original text, giving C∗\mathcal{C}^*. Apart from the added calls, the text of C∗\mathcal{C}^* is exactly the same as C\mathcal{C}, so when the model is fine-tuned on it, it sees the same knowledge as before, and its language ability shouldn't suffer.

The text is a subset of CCNet, and the model is GPT-J 6B. To save compute on annotation, the authors first use rules to pick out text that is more likely to need tools; for the calculator, for example, they only process passages containing at least three numbers. Fine-tuning uses a batch size of 128 and a learning rate of 10−510^{-5}, with linear warmup over the first 10% of steps.

How many calls survive varies by tool. At a filtering threshold of τf=1.0\tau_f = 1.0, Wikipedia search has 60,974, the calendar 20,587, question answering 18,526, translation 1,034, and the calculator only 994.

Inference

After fine-tuning, decoding works as usual. When the model writes "→", it means it's waiting for a tool result. Decoding pauses at that point, the tool actually gets called, the result and "]" are appended, and the model carries on writing. For example, decoding pauses when the model has written "[Calculator(400 / 1400) →"; the calculator returns 0.29, "0.29]" is appended, and the model goes on to write "29%".

Decoding also gets two tweaks. The model is allowed to start a call whenever <API> is among the 10 most likely tokens, not only when it's ranked first, which encourages it to use tools more. And each input gets at most one call, so the model can't get stuck in a loop of calling over and over without ever writing an answer.

Discussion

Experimental setup

The main comparison is among four models: the original GPT-J; GPT-J + CC, fine-tuned on the same CCNet text but without tool calls; Toolformer; and Toolformer with the probability of <API> set to 0, so it can't use tools. There are also comparisons with OPT 66B and GPT-3 175B (davinci, without instruction tuning).

Everything is zero-shot. The task is described in words only, no examples are given, and the model isn't told which tool to use. Scoring is fairly lenient; for example, an answer counts as correct if it appears within the first few words of the model's output.

Results

TaskGPT-JTools offToolformerGPT-3 175B
LAMA SQuAD17.822.133.826.8
LAMA Google-RE4.96.311.57.0
LAMA T-REx31.934.953.539.8
ASDiv7.514.840.414.0
SVAMP5.26.329.410.0
MAWPS9.915.044.019.8
WebQS18.518.926.329.0
Natural Questions12.812.617.722.6
TriviaQA43.946.748.865.9
TempLAMA13.712.716.315.5
Dateset3.95.927.30.8

Compiled from Tables 3, 4, 5 and 7 of the paper; "Tools off" is Toolformer with API calls disabled; values are in %.

Fill-in-the-blank knowledge questions (LAMA): on 98.1% of questions the model goes to the question-answering tool on its own, and it beats GPT-3 on all three subsets. LAMA's sentences come from Wikipedia, so Wikipedia search is disabled here to keep things from being too easy.

Math word problems: the calculator is called on 97.9% of problems, and scores are more than double those with tools off, far above GPT-3 as well. Interestingly, Toolformer with tools off also does better than GPT-J. The authors guess that after seeing lots of calculations and their results, the model's own arithmetic improved too.

Open-domain QA: the question-answering tool was fine-tuned on Natural Questions, so using it would amount to cheating, and it's disabled here. The model turns to Wikipedia search 99.3% of the time instead, beating the other GPT-J models but still losing to GPT-3. The authors see two reasons. The BM25 search engine is very basic and often returns irrelevant results, and Toolformer can't rephrase its query and search again, or look further down the results.

Time-related questions: Dateset can only be answered if you know today's date, as in "What day of the week was it 30 days ago?" The calendar is used on 54.8% of its questions, and the score rises from 5.9% to 27.3%. TempLAMA asks about facts that change over time, such as which team a certain player plays for. Toolformer improves there too, but only 0.2% of questions use the calendar; the gains come mostly from search and question answering. The best approach would be to look up the date first and then ask with that date, but only one call is allowed per question, and the calls in the training data were all sampled separately for each tool, so the model has never seen this kind of relay.

Multilingual QA (MLQA): the questions are in Arabic, German, Spanish, Hindi, Vietnamese or Simplified Chinese, and the context paragraphs are in English. Depending on the language, the model first translates the question into English on 63.8% to 94.9% of questions, and does better than with tools off; Hindi is the exception, with translation used only 7.3% of the time. Toolformer doesn't consistently beat the original GPT-J, though, because fine-tuning on CCNet by itself made some languages worse.

Language ability doesn't get worse

ModelWikiTextCCNet
GPT-J9.910.6
GPT-J + CC10.310.5
Toolformer (tools off)10.310.5

Compiled from Table 8 of the paper; the numbers are perplexity, lower is better.

Fine-tuning on CCNet makes WikiText perplexity slightly worse, but adding tool calls or not makes no difference. Perplexity with tools switched on can't be computed, because any position might call any tool, and getting the probability would mean summing over every possibility.

How big does the model need to be?

Original Figure 4. Average scores on three task groups, LAMA, math and QA, for GPT-2 at 124M, 355M, 775M and 1.6B and for GPT-J after Toolformer training. The blue line is with tools, the red line is with tools off, and the dashed line is GPT-3. For LAMA and math, the smallest few models score about the same with or without tools; from around 775M the blue line pulls clearly above the red, and the gap is still large at the biggest model. For QA, there's a gap of a few percentage points even at the smallest model, and it holds all the way to the biggest.
Original Figure 4, Schick et al., from PDF page 9, arXiv license, scholarly quotation.

The authors redid the experiment with four sizes of GPT-2, giving them only three tools: question answering, the calculator and Wikipedia search. The ability to use tools only appears at around 775M parameters; smaller models do about the same with or without tools. Wikipedia search is the exception, which even small models can make use of; the authors guess that's because this API is simpler. As models grow, they get better without tools and better at using tools at the same time, so the gap between the two never goes away.

The decoding threshold

kkT-RExCall rateWebQSCall rate
034.90.0%18.90.0%
147.840.3%19.38.5%
352.982.8%26.399.3%
1053.598.1%26.3100%

Compiled from Table 9 of the paper, showing selected columns; the T-REx and WebQS columns are accuracy (%).

kk is how high <API> has to rank before a call starts. k=0k = 0 amounts to turning tools off, and k=1k = 1 is ordinary greedy decoding. With ordinary greedy decoding, the model calls a tool on only 8.5% of WebQS questions, and the score barely changes; only when the threshold is relaxed to the top 3 does it start using tools heavily.

At k=1k = 1, on the questions where the model decides not to call, its scores (44.3% on T-REx, 19.9% on WebQS) are actually higher than the average with tools forbidden entirely (34.9% and 18.9%). That shows the model knows, to some degree, which questions it can't handle, and saves the tools for those. Once the threshold is relaxed, this judgment disappears.

What the kept calls look like

The authors list some examples sorted by filter score Li−−Li+L_i^- - L_i^+, from high to low. The high-scoring ones mostly make sense. One searches for Flodden when the text mentions the UK's oldest war memorial window, with a score of 5.49; another asks how long the Nile is when the text reaches "The Nile has an approximate length of", with a score of 2.08.

The low-scoring ones are mostly useless, such as computing 85 / 23 after "85 patients (23%)", with a score of −0.02. But "Catch this fast train to success!" triggers a search for "Fast train success", which turns up a 1971 song with completely unrelated content, yet it still lowers perplexity, with a score of 0.92. The authors think a small amount of this kind of noise isn't necessarily a bad thing, since it keeps the model from blindly trusting every tool result.

Limitations

The authors list these themselves:

  • No chaining tools: for example, looking up the date first and then asking with it. The calls for each tool are sampled separately, so the training data has no such examples.
  • No interaction: a search engine may return hundreds of results, and the model can neither browse them nor revise its query.
  • Sensitive to wording: whether to call a tool is easily swayed by how the input is phrased.
  • Very inefficient: processing more than a million documents yields only a few thousand useful calculator calls. The authors suggest iterating, as other bootstrapping methods do.
  • Ignores cost: different tools have very different computational costs, and the model doesn't take that into account when deciding.

What came after

Four months after Toolformer came out, tool use became a standard feature of commercial models, though by a different route. On June 13, 2023, OpenAI added function calling to its API. Developers describe their functions in JSON; when the model decides it needs one, it outputs a JSON object with the arguments, the application runs it, and the result is handed back to the model. The tools are defined by developers, and the model learns during training to call them from their descriptions, with no need to figure it out from ordinary text on its own.

Chaining and interaction, which Toolformer listed as limitations, became basic capabilities in the agent loop. The model calls a tool, looks at the result and decides the next step, going back and forth dozens of times on a single task. In November 2024, Anthropic introduced the Model Context Protocol as an open standard for connecting AI assistants to all kinds of data sources and tools, and the way tools get hooked up started to converge.

Conclusion

Toolformer gives each tool just a few demonstrations, lets GPT-J insert tool calls into ordinary text on its own, and filters them by "does this make the following text easier to predict?" After fine-tuning, this 6B model beats GPT-3 175B on knowledge fill-in-the-blank, math word problems and date questions, and its language ability doesn't get worse.

The most interesting design choice in this paper is letting the model judge for itself whether a tool helps. Where to insert a question and which answer helps are all decided by the model's loss, with nobody telling it; even the call that turned up that 1971 song was kept because the model itself found it helpful.

Later tool use didn't keep this self-supervision. Instead, developers write the tool descriptions and the model calls tools accordingly, which is simple and controllable. But the format of "write the call into the text, splice the result back into the text, and let the model keep writing" did stick, and every step of today's agents works this way.