Weak quantitative standards in linguistics research.

By Edward Gibson, E. Fedorenko. 2010

123 citations recorded by the source corpus.

A serious methodological weakness affecting much research in syntax and semantics within the field of linguistics is that the data presented as evidence are often not quantitative in nature. In particular, the prevalent method in these fields involves evaluating a single sentence/meaning pair, typically an acceptability judgment performed by just the author of the paper, possibly supplemented by an informal poll of colleagues. Although acceptability judgments are a good dependent measure of linguistic complexity (results from acceptability–judgment experiments are highly systematic across speakers and correlate with other dependent measures, but see Ref.

Subfields: Cognitive Neuroscience, Language and Linguistics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Language and Linguistics

Citations in the corpus

Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments (2019, background): “also attempt to model gradient crowdsourced acceptability judgments, reflecting an ongoing debate about whether binary expert judgments like those in CoLA are reliable (Gibson and Fedorenko, 2010; Sprouse and Almeida, 2012).”
A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001--2010 (2013, background): “…Haider, 2007; Newmeyer, 2007; Sprouse, 2007; Culbertson and Gross, 2009; Myers, 2009a,b; Phillips, 2010; Bader and Häussler, 2010; Dąbrowska, 2010; Gibson and Fedorenko, 2010; Culicover and Jackendoff, 2010; Gross and Culbertson, 2011; Sprouse, 2011b; Weskott and Fanselow, 2011; Gibson et al.,…”
The reliability of acceptability judgments across languages (2018, background): “Gibson et al. (2013) refer to this class as “theoretically meaningful contrasts”. We prefer to use the more neutral term Class III judgments: it is often difficult to assess the theoretical significance of a particular contrast, and it is not clear whether there is a relationship between the “obviou”
SNAP judgments: A small N acceptability paradigm (SNAP) for linguistic acceptability judgments (2016, methodology): “…data have led to further discussions of the weaknesses of the informal method (e.g. Arppe & Järvikivi 2007, Cowart 1997, Featherston 2005, Gibson & Fedorenko 2010, 2013, Gibson et al. 2013, Gross & Culbertson 2011, Lau et al. 2017, Linzen & Oseki 2015, Schütze 1996, Sorace & Keller 2005, Wasow &…”
Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments (2019, methodology): “Early work in this area primarily uses recurrent models like Long Short-Term Memory (Hochreiter and Schmidhuber, 1997, LSTM) networks to reduce variable length sequences into fixed-length sentence embeddings. Current state of the art sentence encoders are pretrained on language modeling or related t”