Decoding and Diversity in Machine Translation
By Nicholas Roberts, Davis Liang, Graham Neubig, Zachary Chase Lipton. 2020
24 citations recorded by the source corpus.
Neural Machine Translation (NMT) systems are typically evaluated using automated metrics that assess the agreement between generated translations and ground truth candidates. To improve systems with respect to these metrics, NLP researchers employ a variety of heuristic techniques, including searching for the conditional mode (vs. sampling) and incorporating various training heuristics (e.g., label smoothing). While search strategies significantly improve BLEU score, they yield deterministic outputs that lack the diversity of human translations. Moreover, search tends to bias the distribution of translated gender pronouns. This makes human-level BLEU a misleading benchmark in that modern MT systems cannot approach human-level BLEU while simultaneously maintaining human-level translation diversity. In this paper, we characterize distributional differences between generated and real translations, examining the cost in diversity paid for the BLEU scores enjoyed by NMT. Moreover, our study implicates search as a salient source of known bias when translating gender pronouns.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Nanyun Peng
- Emily Sheng
- Kai-Wei Chang
- Premkumar Natarajan
- Danielle Saunders
- Bill Byrne
- Rosie Sallis
- Yejin Choi
In Natural Language Processing
- Nanyun Peng
- Kai-Wei Chang
- Emily Sheng
- Premkumar Natarajan
- Danielle Saunders
- Bill Byrne
- Rosie Sallis
- Yejin Choi
Citations in the corpus
- Societal Biases in Language Generation: Progress and Challenges (2021, background): “Despite variations in fluency and diversity between deterministic versus stochastic, search versus sampling procedures, there are limited studies (Roberts et al., 2020) on how different decoding properties affect”
- First the Worst: Finding Better Gender Translations During Beam Search (2021, background): “Roberts et al. (2020) find that beam 078 search amplifies gender bias compared to sampling 079 search.”