How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation

By Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, Hany Hassan Awadalla. 2023

603 citations recorded by the source corpus.

Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (2024, methodology): “The WMT competition winners for each direction as reported in WMT’21 and WMT’22 correspond to those used by Hendy et al. (2023).”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “Finally, there is extensive work on analyzing the translation output of LLMs, generally finding that it is more fluent than accurate (Hendy et al., 2023; Anonymous, 2023), good at handling document context (Wang et al.”
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2023, methodology): “For Common MT, we adopt automatic metrics like COMET5 and BLEURT6, which are widely adopted evaluation metrics for LLM-based translation literature (He et al., 2023; Hendy et al., 2023; Garcia et al., 2023; Pilault et al., 2023).”
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM (2023, neutral): “The authors of (Brown et al., 2020) and (Hendy et al., 2023) also use a verbose, instruction-like prompt in their zero-shot setup, and a much more compact one for few shots experiments.”
A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models (2023, background): “In addition, Yang et al. (2023) and Hendy et al. (2023) investigated the potential of ChatGPT (i.e., gpt-3.”
Low-Resource Languages Jailbreak GPT-4 (2023, result): “More importantly, our results also suggest that researchers have underestimated the capability of the LLMs in understanding and generating text in low-resource languages [4, 18, 22, 39, 46, 48].”
ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification (2023, result): “When Huang et al. (2023) examined the performance of ChatGPT on categorizing implicit hate speech and providing natural language explanations for the implicit hate speech detection, the results showed great potential of ChatGPT for this and similar tasks.”
Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023, background): “This confirms the findings of Hendy et al. (2023) who demonstrated superior translation capabilities with Davinci-003 over all other previous GPT variants.”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “…and state-of-the-art systems for important tasks in NLP such as text summarization (Wang et al., 2023a; Yang et al., 2023), machine translation (Hendy et al., 2023; Jiao et al., 2023; Kocmi and Federmann, 2023), question answering (Tan et al., 2023; Omar et al., 2023; Lai et al., 2023),…”
A Categorical Archive of ChatGPT Failures (2023, background): “However, for future investigations, it may be useful to focus on more specific categories of problems, such as sentiment analysis, named entity recognition, translation [27, 28], summarization [66], and language ambiguity [47], in order to gain a more detailed understanding of ChatGPT’s shortcomings”