What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
By Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang and 2 more. 2022
48 citations recorded by the source corpus.
We present the results of the NLP Community Metasurvey. Run from May to June 2022, it elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split in half on the importance of artificial general intelligence, whether language models understand language, and the necessity of linguistic structure and inductive bias for solving NLP problems. In addition, the survey posed meta-questions, asking respondents to predict the distribution of survey responses. This allows us to uncover false sociological beliefs where the community’s predictions don’t match reality. Among other results, we find that the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Mohit Bansal
- Ethan Perez
- Samuel R. Bowman
- Jianfeng Gao
- Quoc V. Le
In Natural Language Processing
- Ethan Perez
- Sam Bowman
- Irene Solaiman
- Dario Amodei
- Samuel R. Bowman
- T. Henighan
- Amanda Askell
- Dawn Drain
Citations in the corpus
- Discovering Language Model Behaviors with Model-Written Evaluations (2022, background): “We start with a few multiple-choice Q&A datasets where people often disagree about the right answer: politics (Pew Research Center Political Typology Quiz questions2), philosophy (PhilPapers 2020 Survey questions; Bourget and Chalmers, 2020), and Natural Language Processing (NLP Experts Survey quest”
- The Gradient of Generative AI Release: Methods and Considerations (2023, background): “For researchers in natural language processing, whether developers should be ethically responsible for downstream misuse of a publicly released system is contentious, with about half of researchers believing professionals should be responsible[54].”
- The debate over understanding in AI’s large language models (2022, unknown): “%) disagreed [57].”