A Challenge Set and Methods for Noun-Verb Ambiguity

By Ali El-Kahky, Kellie Webster, D. Andor, Emily Pitler. 2018

18 citations recorded by the source corpus.

English part-of-speech taggers regularly make egregious errors related to noun-verb ambiguity, despite having achieved 97%+ accuracy on the WSJ Penn Treebank since 2002. These mistakes have been difficult to quantify and make taggers less useful to downstream tasks such as translation and text-to-speech synthesis. This paper creates a new dataset of over 30,000 naturally-occurring non-trivial examples of noun-verb ambiguity. Taggers within 1% of each other when measured on the WSJ have accuracies ranging from 57% to 75% accuracy on this challenge set. Enhancing the strongest existing tagger with contextual word embeddings and targeted training data improves its accuracy to 89%, a 14% absolute (52% relative) improvement. Downstream, using just this enhanced tagger yields a 28% reduction in error over the prior best learned model for homograph disambiguation for textto-speech synthesis.

Subfields: Artificial Intelligence, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hannaneh Hajishirzi
2. Ellie Pavlick
3. Mohit Bansal
4. Iryna Gurevych
5. Tal Linzen
6. Luke S. Zettlemoyer
7. Akari Asai
8. Samuel R. Bowman

In Natural Language Processing

1. Ellie Pavlick
2. Hannaneh Hajishirzi
3. Iryna Gurevych
4. Tal Linzen
5. Mohit Bansal
6. Akari Asai
7. Luke S. Zettlemoyer
8. Samuel R. Bowman

In Computational Linguistics & Semantics

1. Iryna Gurevych
2. Steven Bedrick
3. Tal Linzen
4. Hannaneh Hajishirzi
5. Ellie Pavlick
6. Akari Asai
7. Kyle Gorman

Citations in the corpus

• Analysis Methods in Neural Language Processing: A Survey (2018, neutral): “(Lake and Baroni, 2018) seq2seq Compositionality English 20910 Automatic (Elkahky et al., 2018) POS tagging Noun-verb ambiguity English 32654 Semi-auto.”
• PAWS: Paraphrase Adversaries from Word Scrambling (2019, neutral): “This has led, for example, to new datasets for noun-verb ambiguity (Elkahky et al., 2018) and gender bias in coreference (Web-ster et al., 2018; Rudinger et al., 2018; Jieyu Zhao, 2018).”
• Logic-Guided Data Augmentation and Regularization for Consistent Question Answering (2020, methodology): “We hypothesize that the baseline model exploits statistical pattern, or dataset bias presented in questions and that our method reduces the model’s tendency to exploit those spurious statistical patterns (He et al., 2019; Elkahky et al., 2018), which leads to large improvements in consistency.”
• Predicting Inductive Biases of Pre-Trained Models (2021, methodology): “…procedures of neural NLP models (Jia & Liang, 2017b; Glockner et al., 2018; Dasgupta et al., 2018; Gururangan et al., 2018; Poliak et al., 2018b; Elkahky et al., 2018; Ettinger et al., 2016; Linzen et al., 2016; Isabelle et al., 2017; Naik et al., 2018; Jia & Liang, 2017a; Linzen et al., 2016;…”
• Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures (2020, methodology): “An alternative approach is to augment the training data with additional counterexamples for the bias (McCoy et al., 2019b; Elkahky et al., 2018).”
• We Need to Talk about Standard Splits (2019, methodology): “English is a somewhat challenging language for POS tagging because of its relatively impoverished inflectional morphology and pervasive noun-verb ambiguity (Elkahky et al., 2018).”