On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

By Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, Shmargaret Shmitchell. 2021

7,969 citations recorded by the source corpus.

The past 3 years of work in NLP have been characterized by the development and deployment of ever larger language models, especially for English. BERT, its variants, GPT-2/3, and others, most recently Switch-C, have pushed the boundaries of the possible both through architectural innovations and through sheer size. Using these pretrained models and the methodology of fine-tuning them for specific tasks, researchers have extended the state of the art on a wide array of tasks as measured by leaderboards on specific benchmarks for English. In this paper, we take a step back and ask: How big is too big? What are the possible risks associated with this technology and what paths are available for mitigating those risks? We provide recommendations including weighing the environmental and financial costs first, investing resources into curating and carefully documenting datasets rather than ingesting everything on the web, carrying out pre-development exercises evaluating how the planned approach fits into research and development goals and supports stakeholder values, and encouraging research directions beyond ever larger language models.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Percy Liang
  4. Jianfeng Gao
  5. Yejin Choi
  6. Quoc V. Le
  7. Graham Neubig
  8. Yoshua Bengio

In Information Systems

  1. Percy Liang
  2. Luke S. Zettlemoyer
  3. Mohit Bansal
  4. Graham Neubig
  5. Samuel R. Bowman
  6. Richard Socher
  7. Matt Gardner
  8. Christopher D. Manning

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Yejin Choi
  3. Hinrich Schütze
  4. Noah A. Smith
  5. Timo Schick
  6. Jianfeng Gao
  7. Mohit Bansal
  8. Hannaneh Hajishirzi

In Safety, Robustness & Interpretability

  1. Yulia Tsvetkov
  2. Adina Williams
  3. Nicholas Carlini
  4. Percy Liang
  5. Kai-Wei Chang
  6. Yejin Choi
  7. Eric Wallace
  8. Eric Michael Smith

Citations in the corpus