Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine
By Wenxiang Jiao, Wenxuan Wang, Jen-Tse Huang, Xing Wang, Zhaopeng Tu. 2023
408 citations recorded by the source corpus.
This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well with minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language. Further, we explore an interesting strategy named $\mathbf{pivot~prompting}$ for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted, becoming comparable to commercial translation products, even for distant languages. Human analysis on Google Translate and ChatGPT suggests that ChatGPT with GPT-3.5 tends to generate more hallucinations and mis-translation errors while that with GPT-4 makes the least errors. In other words, ChatGPT has already become a good translator. Please refer to our Github project for more details: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator
Subfields: Artificial Intelligence, Health Informatics
View Paper: https://www.semanticscholar.org/paper/780c99d13537370f63c03feeb1343bed9d98a4f9
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Health Informatics
Citations in the corpus
- Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “In reference to study [37], ChatGPT was compared to three commercial translation products: Google Translate2, DeepL Translate3, and Tencent TranSmart4.”
- GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, methodology): “We follow recent successes to utilize the GPT-4’s strong ability (Jiao et al., 2023)”
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024, background): “The imbalance in pre-training data can lead to imbalances during model training (Liu et al., 2024), resulting in biases against certain demographic groups, such as different genders (Wan et al., 2023), ages (Macnicol, 2006), and various languages (Jiao et al., 2023; Bang et al., 2023).”
- A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, background): “Based on the above technological developments, in the year of 2022, various of AIGC products were developed and iterated by the world’s leading technology companies, for example DALL-E-2 of OpenAI [14] can generate images of high-quality with giving specific descriptions and Meta proposes Make-A-Vid”
- Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, methodology): “In reference to study [34], ChatGPT was compared to three commercial translation products: Google Translate2, DeepL Translate3, and Tencent TranSmart4.”
- ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “…the United States Medical Licensing Examination (Kung et al., 2022) and real exams in a law school (Choi et al., 2023), performing competitively with commercial translation services for some high-resource languages (Jiao et al., 2023), and even producing code from natural language instructions.”