Neural Network Acceptability Judgments

By Alex Warstadt, Amanpreet Singh, Samuel R. Bowman. 2018

1,665 citations recorded by the source corpus.

Abstract This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “CoLA The Corpus of Linguistic Acceptability is a binary single-sentence classification task, where the goal is to predict whether an English sentence is linguistically “acceptable” or not (Warstadt et al., 2018).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “CoLA The Corpus of Linguistic Acceptability (Warstadt et al., 2018) consists of English acceptability judgments drawn from books and journal articles on linguistic theory.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…comprise a collection of text classification tasks meant to test general language understanding abilities: • Sentence acceptability judgment (CoLA [Warstadt et al., 2018]) • Sentiment analysis (SST-2 [Socher et al., 2013])…”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “(6)The datasets are: CoLA (Warstadt et al., 2018), Stanford Sentiment Treebank (SST) (Socher et al.”
Adversarial Training for Large Neural Language Models (2020, background): “As shown in Table 9, it includes question answering (Rajpurkar et al., 2016), linguistic acceptability (Warstadt et al., 2018), sentiment analysis (Socher et al., 2013), text similarity (Cer et al., 2017), paraphrase detection (Dolan and Brockett, 2005), and natural language inference (NLI) (Da-gan…”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “Tables 3 and 2 compares the performance of BART with several recent approaches on the well-studied SQuAD and GLUE tasks (Warstadt et al., 2018; Socher et al., 2013; Dolan & Brockett, 2005; Agirre et al., 2007; Williams et al., 2017; Dagan et al., 2006; Levesque et al., 2011).”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “A.4 DOWNSTREAM EVALUATION TASKS GLUE GLUE is comprised of 9 tasks, namely Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018), Stanford Sentiment Treebank (SST; Socher et al., 2013), Microsoft Research Paraphrase Corpus (MRPC; Dolan & Brockett, 2005), Semantic Textual Similarity…”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, background): “The Corpus of Linguistic Acceptability is a task to predict whether an English sentence is a grammatically correct one (Warstadt et al., 2018).”
Multi-Task Deep Neural Networks for Natural Language Understanding (2019, background): “CoLA The Corpus of Linguistic Acceptability is to predict whether an English sentence is linguistically acceptable or not (Warstadt et al., 2018).”
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “CoLA and SST are again the exceptions: Both tasks deteriorated moderately with supplementary training, and CoLA trained with the auxiliary language modeling objective in particular showed highly unstable results when trained on small amounts of data.”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, neutral): “We follow the entity masking schema from Zhang et al. (2017) GLUE The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classiﬁcation tasks: • Two sentence-level classiﬁcation tasks including CoLA (Warstadt et al., 2018) for evaluating lingui”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, background): “…is a collection of nine language understanding tasks as in Table 8, including question answering (Rajpurkar et al., 2016), linguistic acceptability (Warstadt et al., 2018), sentiment analysis (Socher et al., 2013), text similarity (Cer et al., 2017), paraphrase detection (Dolan and Brockett,…”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “(6)The Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018) consists of examples of expert English sentence acceptability judgments drawn from 22 books and journal articles on linguistic theory.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “Experiments For task sampling, we run experiments with multitask learning on the full set of nine GLUE tasks, as well as three subsets: single sentence tasks (S1: SST, CoLA), similarity and paraphrase tasks (S2: MRPC, STS, QQP), and inference tasks (S3: WNLI, QNLI, MNLI, RTE).”
Finetuned Language Models Are Zero-Shot Learners (2021, neutral): “Linguistic acceptability: CoLA (Warstadt et al., 2019) 5.”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, neutral): “• BoolQ (Clark et al., 2019) • CoLA (Warstadt et al., 2019) • MNLI (Williams et al., 2018) • MRPC (Dolan and Brockett, 2005) • QNLI (Rajpurkar et al., 2016; Wang et al., 2019b) • RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) • SST-2 (Socher et al., 20”