Can GPT-3 Pass a Writer’s Turing Test?
By Katherine Elkins, Jon Chun. 2020
187 citations recorded by the source corpus.
Until recently the field of natural language generation relied upon formalized grammar systems, small-scale statistical models, and lengthy sets of heuristic rules. This older technology was fairly limited and brittle: it could remix language into word salad poems or chat with humans within narrowly defined topics. Recently, very large-scale statistical language models have dramatically advanced the field, and GPT-3 is just one example. It can internalize the rules of language without explicit programming or rules. Instead, much like a human child, GPT-3 learns language through repeated exposure, albeit on a much larger scale. Without explicit rules, it can sometimes fail at the simplest of linguistic tasks, but it can also excel at more difficult ones like imitating an author or waxing philosophical.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Large Language Models, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Ronan Le Bras
- Maarten Sap
- Daniel Fried
- Noah A. Smith
- Alessandro Lenci
- Mohit Bansal
- Luke S. Zettlemoyer
In Computational Theory and Mathematics
- Alessandro Lenci
- Massimo Chiriatti
- Li Lucy
- Yejin Choi
- Luciano Floridi
- David Bamman
- Ronan Le Bras
- Daniel Fried
In Large Language Models
- Alessandro Lenci
- Yejin Choi
- Maarten Sap
- Li Lucy
- David Bamman
- Ronan Le Bras
- Daniel Fried
- Tuhin Chakrabarty
In Natural Language Generation
Citations in the corpus
- Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “ever, increasing scrutiny has shed light on the shortcomings of these LLMs, showing that they often fall prey to spurious correlational patterns instead of displaying higher-order reasoning (Elkins and Chun, 2020; Dale, 2021; Marcus, 2022).”
- GPT-3: Its Nature, Scope, Limits, and Consequences (2020, background): “Despite its mathematical, sematic and ethical shortcomings—or better, despite not being designed to deal with mathematical, semantic, and ethical questions—GPT-3 writes better than many people (Elkins and Chun 2020).”
- Gender and Representation Bias in GPT-3 Generated Stories (2021, methodology): “thus has potential for wide use in narrative generation tasks (Brown et al., 2020; Brockman et al., 2020; Scott, 2020; Elkins and Chun, 2020; Branwen, 2020).”
- Help me write a Poem - Instruction Tuning as a Vehicle for Collaborative Poetry Writing (2022, background): “This has spurred increasing interest in computer-assisted creativity (See et al., 2019; Elkins and Chun, 2020; Ramesh et al., 2022; Branwen, 2020) such as building co-creative assistants for writing stories, poems, and argumentative essays (Lee et al., 2022; Swan-son et al., 2021; Uthus et al.,…”
- Understanding Natural Language Understanding Systems. A Critical Analysis (2023, background): “They give us the illusion of «talking like us», an illusion that, we must admit, is nowadays extremely realistic, so that they have de facto made the classical Turing Test based on the «imitation game» obsolete or at least ineffective (Elkins and Chun 2020).”
- ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing (2023, background): “Also, none that are close to achieving GPT-3's capabilities are as widely available to the public (Nature Machine Intelligence Editorial Board, 2020; Liu et al., 2019; Liu et al., 2022; Yang et al., 2019; Elkins & Chun, 2020).”