A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars

By Ezra Black, Steven P. Abney, D. Flickenger, Claudia Gdaniec, Ralph Grishman, Philip Harrison, Donald Hindle, Robert Ingria, Frederick Jelinek, Judith L. Klavans and 5 more. 1991

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Call for Clarity in Reporting BLEU Scores (2018, methodology): “An instructive comparison is the PARSEVAL metric for computing parser accuracy (Black et al., 1991).”
The Penn Treebank: Annotating Predicate Argument Structure (1994, background): “The ongoing effort [1] to develop a standard objective methodology to compare parser outputs across widely divergent grammatical frameworks has now resulted in a widely supported standard for parser comparison.”
Unbounded Dependency Recovery for Parser Evaluation (2009, background): “Finally, some researchers have argued that the Parseval metrics (Black et al., 1991) are too forgiving with respect to certain errors and that an evaluation based on syntactic dependencies, for which scores are typically lower, is a better test of parser performance (Lin, 1995; Carroll et al.,…”
The Stanford Typed Dependencies Representation (2008, background): “Nevertheless, we agree with the widespread sentiment that dependency-based evaluation of parsers avoids many of the problems of the traditional Parseval measures (Black et al., 1991), and to the extent that the Stanford dependency representation is an effective representation for the tasks envisione”
Judging Grammaticality: Experiments in Sentence Classification (2013, methodology): “These features include the difference in logarithmic parse probability between the trees and structural differences between the trees measured using various parser evaluation metrics (Black et al., 1991; Sampson & Babarczy, 2003).”
The Penn Chinese TreeBank: Phrase structure annotation of a large corpus (2005, methodology): “This is done in three steps: ﬁrst, we calculate inter-annotator agreement by running the Parseval software 18 that produces three metrics – precision, recall and numbers of crossing brackets (Black et al. 1991).”
Spoken Language Derived Measures for Detecting Mild Cognitive Impairment (2011, methodology): “5We use the standard PARSEVAL measures [37] for evaluation of parse accuracy: labeled precision is the number of correct labeled spans divided by labeled spans in the automatic parse; labeled recall is the number of correct labeled spans divided by labeled spans in the true parse; and F-measure is t”
Introducing the SPMRL 2014 Shared Task on Parsing Morphologically-rich Languages (2014, methodology): “As a consequence of the raw scenario, it was not possible to (only) rely on the accepted parsing metrics, labeled bracket evaluation via EVALB1 (Black et al., 1991), Leaf-Ancestor (Sampson and Babarczy, 2003) for constituents and CONLL X’s Labeled/Unlabeled Attachment Score for dependencies (Buchhol”
Sentence Level Discourse Parsing using Syntactic and Lexical Information (2003, methodology): “The performance is assessed us-ing labeled recall and labeled precision as de(cid:2)ned by the standard Parseval metric (Black et al., 1991).”