The Turking Test: Can Language Models Understand Instructions?

By Avia Efrat, Omer Levy. 2020

105 citations recorded by the source corpus.

Supervised machine learning provides the learner with a set of input-output examples of the target task. Humans, however, can also learn to perform new tasks from instructions in natural language. Can machines learn to understand instructions as well? We present the Turking Test, which examines a model's ability to follow natural language instructions of varying complexity. These range from simple tasks, like retrieving the nth word of a sentence, to ones that require creativity, such as generating examples for SNLI and SQuAD in place of human intelligence workers ("turkers"). Despite our lenient evaluation methodology, we observe that a large pretrained language model performs poorly across all tasks. Analyzing the model's error patterns reveals that the model tends to ignore explicit instructions and often generates outputs that cannot be construed as an attempt to solve the task. While it is not yet clear whether instruction understanding can be captured by traditional language models, the sheer expressivity of instruction understanding makes it an appealing alternative to the rising few-shot inference paradigm.

Subfields: Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “Efrat & Levy (2020) evaluated GPT-2 (Radford et al., 2019) on simple tasks ranging from retrieving the nth word of a sentence to generating examples for SQuAD, concluding that GPT-2 performs poorly across all tasks.”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, background): “However, the resulting data is limited to existing academic benchmarks, even though the instruction paradigm can describe any text-based task (Efrat and Levy, 2020).”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “The closest work to ours is Efrat and Levy (2020) who also examine models’ ability to follow the crowdsourcing instructions that were used to build existing datasets.”
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021, background): “This is also remotely relevant to the literature on learning from instructions which expect a model to adapt its behavior according declarative instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021).”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “Prior work has found it promising to train the model that reads the natural language description of the task (called instructions) and performs a new task at inference (Mishra et al., 2021b; Efrat and Levy, 2020; Wei et al., 2022a; Sanh et al., 2022).”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “…motivated by building models that are generalizable across a variety of NLP tasks, when prompted with either a few examples (Ye and Ren, 2021; Bragg et al., 2021) or language deﬁnitions (Efrat and Levy, 2020; Weller et al., 2020; Zhong et al., 2021; Mishra et al., 2022b,a; Parmar et al., 2022).”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, methodology): “, instruction [109], such as PAGeR [415], ConTinTin [477] and ENTAILMENT [460].”
SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model (2023, background): “There are two major strategies to leverage the pre-trained knowledge [42]: (1) fine-tune the representations, or (2) keep the representations fixed while learning a few additional parameters ( e.g ., adaptor [16], prompt [24], instruction [8], etc.).”
ConTinTin: Continual Learning from Task Instructions (2022, neutral): “For example, Efrat and Levy (2020) tested GPT-2 (Radford et al., 2019) to understand instructions like “ listing nouns ”, “ output the n th word or char ” and real-world MTurk instructions to anno-tate some popular datasets.”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “Engineering instructional prompts1 (Efrat and Levy, 2020) has produced successful results in several NLP tasks such as classification (Jiang et al.”
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, background): “The turking test (Efrat and Levy, 2020) was proposed to measure the ef-ﬁcacy of models to follow instructions.”
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions (2021, methodology): “The instruction paradigm (Goldwasser and Roth, 2014; Efrat and Levy, 2020) is a strict generalization of the current instance-only paradigm.”
Selective Annotation Makes Language Models Better Few-Shot Learners (2022, methodology): “…proposed methods to improve in-context learning in many aspects: e.g., meta-training (Chen et al., 2022; Min et al., 2022b), task instructions (Efrat & Levy, 2020; Mishra et al., 2022; Wei et al., 2021; Sanh et al., 2022), or task formulation (Holtzman et al., 2021; Zhao et al., 2021; Min et…”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “…few-shot approaches ﬁnd that current LMs are mostly unable to understand complex instructions that go beyond short prompts or simple questions (Efrat and Levy, 2020; Weller et al., 2020; Webson and Pavlick, 2021) and that they are highly sensitive to the exact wording of the instructions…”
LIMA: Less Is More for Alignment (2023, background): “These ﬁgures place GPT-4 on-par in agreement with human annotators, essentially passing the Turking Test for this task [Efrat and Levy, 2020].”
HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models (2022, background): “Instructions, proposed as an extension to prompts, describe tasks in natural language (Efrat and Levy, 2020; Weller et al., 2020) and guide models to generalize to unseen tasks (Mishra et al., 2022b; Wei et al., 2021; Ouyang et al., 2022; Sanh et al., 2021; with 4 questions (poem and dialogue…”