Toolformer: Language Models Can Teach Themselves to Use Tools

By Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke S. Zettlemoyer, Nicola Cancedda, Thomas Scialom. Neural Information Processing Systems, 2023

4,753 citations recorded by the source corpus.

Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel. In this paper, we show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. This is done in a self-supervised way, requiring nothing more than a handful of demonstrations for each API. We incorporate a range of tools, including a calculator, a Q\&A system, two different search engines, a translation system, and a calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.

Subfields: Artificial Intelligence, Large Language Models, LLM Agents & Tool Use

View Paper: https://arxiv.org/abs/2302.04761

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Graham Neubig
  6. Quoc V. Le
  7. Yejin Choi
  8. Caiming Xiong

In Large Language Models

  1. Jianfeng Gao
  2. Luke S. Zettlemoyer
  3. Zhiyuan Liu
  4. Graham Neubig
  5. Weizhu Chen
  6. Hinrich Schütze
  7. Hannaneh Hajishirzi
  8. Danqi Chen

In LLM Agents & Tool Use

  1. Stephen Marcus McAleer
  2. Pierre Baldi
  3. Geunwoo Kim
  4. Aleksandra Faust
  5. Izzeddin Gur
  6. Panupong Pasupat
  7. Mustafa Safdari
  8. Kevin Qinghong Lin

Citations in the corpus