Prompting Large Language Model for Machine Translation: A Case Study

By Biao Zhang, Barry Haddow, Alexandra Birch. International Conference on Machine Learning, 2023

424 citations recorded by the source corpus.

Research on prompting has shown excellent performance with little or even no supervised training across many tasks. However, prompting for machine translation is still under-explored in the literature. We fill this gap by offering a systematic study on prompting strategies for translation, examining various factors for prompt template and demonstration example selection. We further explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning in prompting. Extensive experiments with GLM-130B (Zeng et al., 2022) as the testbed show that 1) the number and the quality of prompt examples matter, where using suboptimal examples degenerates translation; 2) several features of prompt examples, such as semantic similarity, show significant Spearman correlation with their prompting performance; yet, none of the correlations are strong enough; 3) using pseudo parallel prompt examples constructed from monolingual data via zero-shot prompting could improve translation; and 4) improved performance is achievable by transferring knowledge from prompt examples selected in other settings. We finally provide an analysis on the model outputs and discuss several problems that prompting still suffers from.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c879413103f8950bdd414c7f60a39bd7748c9be8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “This task is akin to machine translation, where LLMs have already demonstrated impressive performance [89].”
In-context Examples Selection for Machine Translation (2022, background): “These results open space for future research to evaluate the potential of generating diverse and style-speciﬁc outputs for MT.”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “, 2023; Karpinska and Iyyer, 2023) but also prone to problems such as hallucination (Zhang et al., 2023; Guerreiro et al., 2023), and frequently sub-par in low-resource settings (Zhu et al.”
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation (2023, background): “Recently, Zhang et al. (2023) use GLM-130B and show consistent but rather low correlation between the performance of MT and shot selection as compared to random.”
Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models (2023, neutral): “For example, BingChat combines GPT models with web search engines, enabling the provision of summarizations and answers to online content.”
Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study (2023, background): “These LLMs are designed to process and generate human-like languages, enabling them to perform tasks such as language translation [1], content generation [2], conversational AI [3], etc.”
Tool learning with large language models: a survey (2024, neutral): “However, despite their impressive capabilities, LLMs often struggle with complex computations and delivering accurate, timely information due to their reliance on fixed and parametric knowledge [12,13].”