Prompting PaLM for Translation: Assessing Strategies and Performance

By David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, George F. Foster. 2022

234 citations recorded by the source corpus.

Large language models (LLMs) that have been trained on multilingual but not parallel text exhibit a remarkable ability to translate between languages. We probe this ability in an in-depth study of the pathways language model (PaLM), which has demonstrated the strongest machine translation (MT) performance among similarly-trained LLMs to date. We investigate various strategies for choosing translation examples for few-shot prompting, concluding that example quality is the most important factor. Using optimized prompts, we revisit previous assessments of PaLM’s MT capabilities with more recent test sets, modern MT metrics, and human evaluation, and find that its performance, while impressive, still lags that of state-of-the-art supervised systems. We conclude by providing an analysis of PaLM’s MT output which reveals some interesting properties and prospects for future work.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yi Tay
2. Mohit Bansal
3. Quoc V. Le
4. Yoshua Bengio
5. Zhe Gan
6. Graham Neubig
7. Kyunghyun Cho
8. Dacheng Tao

Citations in the corpus

• PaLM 2 Technical Report (2023, methodology): “In this section, we evaluate sentence-level translation quality using recommended practices for high-quality machine translation (Vilar et al., 2022), and measure potential misgendering harms from translation errors.”
• GPT-NER: Named Entity Recognition via Large Language Models (2023, methodology): “, 2022) have obtained significant performance boosts on a variety of natural language processing tasks (Hegselmann et al., 2022; Vilar et al., 2022; Perez et al., 2021; Pietrzak et al., 2021; Wei et al., 2021).”
• How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation (2023, background): “This is consistent with similar findings from (Vilar et al., 2022) on PALM-540B model.”
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023, background): “One of the interesting properties of large language models (LLMs) such as GPT (Brown et al., 2020b) is their (implicit) support for multilingual Q&A. Prompting the model in the right way allows us to translate text between languages (Vilar et al., 2022).”
• Towards Making the Most of ChatGPT for Machine Translation (2023, result): “Although our proposed TSP consistently improves the performance in terms of semantic metric, i.e., COMTE, notably, we have not consistently bridged the task gap in terms of lexical metrics (BLEU and ChrF), which is consistent with similar findings from Vilar et al. (2022) on PALM-540B model.”