Tsnlp | Test Suites for Natural Language Processing

By S. Lehmann, Stephan Oepen, Sylvie Regnier-Prost, K. Netter, Veronika Lux, Judith Klein, Kirsten Falkedal, F. Fouvry, Dominique Estival, Eva Dauphin and 4 more

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “…to benchmarking (further details in Appendix A): • The systematic development of test items (see Appendix A.1), in the form of testsuites [e.g. Lehmann et al., 1996, Ribeiro et al., 2020], audits [e.g. Buolamwini and Gebru, 2018], and adversarial testing [e.g. Ettinger et al., 2017, Niven and…”
The Power of the TSNLP: Lessons from a Diagnostic Evaluation of a Broad-Coverage Parser (2000, methodology): “The TSNLP project produced much insight into the nature of test suite design and delivered three large, publicly available, multi-purpose test suites in English, French, and German [13].”
A Linguistic Evaluation of Rule-Based, Phrase-Based, and Neural MT Engines (2017, methodology): “In brief, it contains segments selected from various parallel corpora and drawn from other sources such as grammatical resources, e.g., the TSNLP Grammar Test Suite (Lehmann et al., 1996) and online lists of typical translation errors.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, unknown): “Following work in NLP [196] and machine translation [178], a number of such suites have been manually constructed to evaluate neural machine translation performance on a range of linguistic phenomena [23, 49, 161].”
On Integrating Discourse in Machine Translation (2017, background): “In a more general vein, Lehmann et al. (1996) develop test suites for NLP in their Test Suites For Natural Language Processing work, for the general evaluation of NLP systems.”