Unbounded Dependency Recovery for Parser Evaluation

By Laura Rimell, Stephen Clark, Mark Steedman. 2009

116 citations recorded by the source corpus.

This paper introduces a new parser evaluation corpus containing around 700 sentences annotated with unbounded dependencies, from seven different grammatical constructions. We run a series of off-the-shelf parsers on the corpus to evaluate how well state-of-the-art parsing technology is able to recover such dependencies. The overall results range from 25% accuracy to 59%. These low scores call into question the validity of using Parseval scores as a general measure of parsing capability. We discuss the importance of parsers being able to recover unbounded dependencies, given their relatively low frequency in corpora. We also analyse the various errors made on these constructions by one of the more successful parsers.

Subfields: Artificial Intelligence, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Linguistics & Semantics

Citations in the corpus

Adversarial Examples for Evaluating Reading Comprehension Systems (2017, background): “However, models can succeed in this paradigm by recognizing patterns that happen to be predictive on most of the test examples, while ignoring deeper, more difficult phenomena (Rimell et al., 2009; Paperno et al., 2016).”
Probing for semantic evidence of composition by means of simple classification tasks (2016, background): “We believe that incorporation of such long-distance dependencies is critical for assessing whether systems are accurately capturing semantic roles across a range of naturally-occurring sentence structures (Rimell et al., 2009; Bender et al., 2011).”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, neutral): “Our work suggests an alternative strategy—evaluation on naturally occurring sentences that are sampled based on their grammatical complexity—which can provide more nuanced tests of language models (Rimell et al., 2009; Bender et al., 2011).”
Stress Test Evaluation for Natural Language Inference (2018, methodology): “Prior work (Rimell et al., 2009; Schneider et al., 2017) performed analyses of model errors in dependency parsing and information extraction.”
Targeted Syntactic Evaluation of Language Models (2018, background): “…Materials can be found at https://github.com/ BeckyMarvin/LM_syneval. lenging syntactic constructions have been constructed for parser evaluation (Rimell et al., 2009; Nivre et al., 2010; Bender et al., 2011), and minimal pair approaches have been proposed for evaluating image captioning…”
Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task (2017, background): “…datasets that exhibit and are annotated for particular linguistic phenomena, in order to facilitate ﬁne-grained analysis of the linguistic capacities of systems for tasks such as parsing, entailment, and semantic relatedness (Rimell et al., 2009; Bender et al., 2011; Marelli et al., 2014).”
Assessing Composition in Sentence Vector Representations (2018, background): “Our focus on assessing linguistically-motivated information relates to work on evaluations that aim for ﬁne-grained analysis of systems’ linguistic capacities (Rimell et al., 2009; Bender et al., 2011; Marelli et al., 2014).”
Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering (2018, background): “es to expose model weaknesses, which provides a diverse set of phenomena (from paraphrases to multi-hop reasoning) that models can’t solve. Other work has explored speciﬁc limitations of NLP systems. Rimell et al. (2009) show that parsers struggle on test examples with unbounded dependencies. The mo”
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (2018, background): “…models are able to achieve good performance under some aggregate metric by overfitting to patterns that are predictive in most cases, often at the expense of more difficult, infrequent instances that require deeper language understanding abilities (Rimell et al., 2009; Jia and Liang, 2017).”
Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus (2011, background): “For the two long-distance phenomena that overlap with those studied in Rimell et al. (2009), our results are comparable.”
Cambridge: Parser Evaluation Using Textual Entailment by Grammatical Relation Comparison (2010, background): “It is known that parsers have difﬁculty with unbounded dependencies (Rimell et al., 2009, from which the un-bounded examples in this dataset were obtained), so this result is not surprising.”
A Gold Standard Dependency Corpus for English (2014, background): “Table 2 compares the performance of the Stanford parser and converter (version 1.6.1) reported in Rimell et al. (2009) with its current performance.”
Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions (2018, background): “Rimell et al. (2009) show that parsers struggle on test examples with unbounded dependencies.”