In-context Examples Selection for Machine Translation
By Sweta Agrawal, Chunting Zhou, Michael Lewis, Luke S. Zettlemoyer, Marjan Ghazvininejad. 2022
260 citations recorded by the source corpus.
Large-scale generative models show an impressive ability to perform a wide range of Natural Language Processing (NLP) tasks using in-context learning, where a few examples are used to describe a task to the model. For Machine Translation (MT), these examples are typically randomly sampled from the development dataset with a similar distribution as the evaluation set. However, it is unclear how the choice of these in-context examples and their ordering impacts the output translation quality. In this work, we aim to understand the properties of good in-context examples for MT in both in-domain and out-of-domain settings. We show that the translation quality and the domain of the in-context examples matter and that 1-shot noisy unrelated example can have a catastrophic impact on output quality. While concatenating multiple random examples reduces the effect of noise, a single good prompt optimized to maximize translation quality on the development dataset can elicit learned information from the pre-trained language model. Adding similar examples based on an n-gram overlap with the test source significantly and consistently improves the translation quality of the outputs, outperforming a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Philipp Koehn
- Felix Stahlberg
- Colin Cherry
- Dacheng Tao
- George F. Foster
- Rico Sennrich
- Markus Freitag
- David Vilar
In Computer Vision and Pattern Recognition
- Bo Du
- David Vilar
- Colin Cherry
- George F. Foster
- Dacheng Tao
- Felix Stahlberg
- Markus Freitag
- Amr Sharaf
In Machine Translation
Citations in the corpus
- Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT (2023, methodology): “We attribute it to the lower correlation between the randomly sampled in-context example and test data, as the prior work (Agrawal et al., 2022) shows that the 1-shot noisy unrelated example could have a”
- Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine (2023, methodology): “With modern techniques like in-context learning with demonstrations (Brown et al., 2020; Agrawal et al., 2022), the translation performance could be further improved.”
- Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “A number of papers (Agrawal et al., 2022; Zhang et al., 2023; Jiao et al., 2023; Hendy et al., 2023) investigate prompt quality and source proximity using methods similar to ours but with different LLMs, notably GPT-3.”
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “[9], Information Extraction [10]–[12], and Machine Translation [13], [14].”
- How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation (2023, methodology): “In the same vein, Agrawal et al. (2022) use a much smaller model XGLM-7.”