GPT-3: Its Nature, Scope, Limits, and Consequences

By Luciano Floridi, Massimo Chiriatti. 2020

2,508 citations recorded by the source corpus.

In this commentary, we discuss the nature of reversible and irreversible questions, that is, questions that may enable one to identify the nature of the source of their answers. We then introduce GPT-3, a third-generation, autoregressive language model that uses deep learning to produce human-like texts, and use the previous distinction to analyse it. We expand the analysis to present three tests based on mathematical, semantic (that is, the Turing Test), and ethical questions and show that GPT-3 is not designed to pass any of them. This is a reminder that GPT-3 does not do what it is not supposed to do, and that any interpretation of GPT-3 as the beginning of the emergence of a general form of artificial intelligence is merely uninformed science fiction. We conclude by outlining some of the significant consequences of the industrialisation of automatic and cheap production of good, semantic artefacts.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Large Language Models

Citations in the corpus

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Because LLMs are designed to grasp the form of language, rather than the meaning [8], they can struggle on tasks like sequential arithmetic problems, multi-hop question answering, recognizing and comparing sentences, or those that require branching logic [10, 12, 28, 70, 71].”
DriveLM: Driving with Graph Visual Question Answering (2023, background): “Embodied Planning with LLMs. Recent work endeavors to leverage the formidable reasoning and generalization capacity of LLMs [20, 36, 68] for embodied AI systems [18, 27, 29, 33, 42, 57, 85].”
On the Gap between Adoption and Understanding in NLP (2021, background): “…we realize that different studies have pointed out that GPT-3 is not exhibiting what we as humans would call “intelligence” (Marcus and Davis, 2020; Floridi and Chiriatti, 2020) and have suggested that learning just from text and without including external experience might be a difﬁcult task…”
GPT (Generative Pre-Trained Transformer)— A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions (2023, background): “Moreover, GPTs [118] can generate personalized content based on customer data, such as search history and past purchases.”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, methodology): “GPT-3 : GPT-3 uses Nvidia A100 GPUs to pre-train on a large 300 billion token set, generating around 175 billion parameters [65].”
A study of generative large language model for medical research and healthcare (2023, positive): “We inherited the GPT-3 architecture implemented in the MegaTron-LM 40 and trained GatorTronGPT models from scratch with the default GPT-3 loss function 13 .”
Understanding Natural Language Understanding Systems. A Critical Analysis (2023, background): “Floridi and Chiriatti (2020) claim that «the real point about AI is that we are increasingly decoupling the ability to solve a problem effectively – as regards the final goal – from any need to be intelligent to do so» (p. 683).”
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024, neutral): “Large language models (LLMs) show remarkable performance on multiple natural language processing (NLP) tasks (Touvron et al., 2023; Ouyang et al., 2022).”
Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering (2024, background): “Common flows, that are suitable for nat-ural language tasks, may not be optimal for code-generation tasks, which include an untapped potential - repeatedly running the generated code, and validating it against known examples.”
The Ghost in the Machine has an American accent: value conflict in GPT-3 (2022, background): “Despite the impressive capabilities of this language model, multiple sources[1-3] have shown the model to be capable of generating toxic or harmful outputs in many areas linked to human values such as gender, race, and ideology.”
FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark (2021, methodology): “In NLP, there are also some methods considering the few shot learning process and achieved some good performance in standard benchmark (Floridi and Chiriatti, 2020).”
MegaScale: Scaling Large Language Model Training to More Than 10, 000 GPUs (2024, background): “For example, GPT-3 [4] has 175 billion parameters and PaLM [5] has 540 billion parameters.”
GPT-3 and InstructGPT: technological dystopianism, utopianism, and “Contextual” perspectives in AI ethics and industry (2022, methodology): “With few or no user-supplied examples, GPT-3 can be used to fulfil language tasks such as summarising text, answering questions, translating, or generating ideas, computer code, novels, and news articles [21].”
Ethics, Rules of Engagement, and AI: Neural Narrative Mapping Using Large Transformer Language Models (2022, background): “GPT models have been shown to generate text outputs often indistinguishable from that of humans [9].”
Auditing large language models: a three-layered approach (2023, background): “Language models start from a source input, called the prompt, to generate the most likely sequences of words, code, or other data [26].”