Is ChatGPT A Good Translator? A Preliminary Study

By Wenxiang Jiao, Wenxuan Wang, Jen-Tse Huang, Xing Wang, Zhaopeng Tu. arXiv.org, 2023

438 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/384e058c262e7f29b50cc4feeb0bd731f6e5377e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Categorical Archive of ChatGPT Failures (2023, background): “However, for future investigations, it may be useful to focus on more specific categories of problems, such as sentiment analysis, named entity recognition, translation [27, 28], summarization [66], and language ambiguity [47], in order to gain a more detailed understanding of ChatGPT’s shortcomings”
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM (2023, methodology): “Finally note the preliminary evaluation of C HAT - GPT in (Jiao et al., 2023), which reports interesting insights regarding the multilingual abilities of this model, as well as proposing innovative techniques to generate (artificial) prompts and to use pivoting in prompting.”
Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT (2023, unknown): “Considering the limits of testing ChatGPT, we follow Jiao et al. (2023) and randomly sample a subset of the dev set as the evaluation data for each task.”
Defending ChatGPT against jailbreak attack via self-reminders (2023, unknown): “The remarkable success of ChatGPT 1 spans a wide range of applications, and it has amassed a rapidly expanding user base 2–4 .”
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (2023, background): “shown remarkable performance on general language tasks (Jiao et al., 2023; Wu et al., 2023; Bang et al., 2023) but still struggle on complex reasoning tasks (Zhu et al.”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “A number of papers (Agrawal et al., 2022; Zhang et al., 2023; Jiao et al., 2023; Hendy et al., 2023) investigate prompt quality and source proximity using methods similar to ours but with different LLMs, notably GPT-3.”
Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (2024, background): “…shown remarkable efficacy in various NLP tasks, which attracts the interest of developing machine translation with these decoder-only LLMs. Recent studies (Zhu et al., 2023a; Jiao et al., 2023b; Hendy et al., 2023; Kocmi et al., 2023; Freitag et al., 2023) indicate that larger LLMs such as GPT-3.”
“Chatting with ChatGPT”: Analyzing the factors influencing users' intention to Use the Open AI's ChatGPT using the UTAUT model (2023, background): “Firstly, irrespective of their discipline, available limited studies [18 – 21] and anecdotal articles [22 – 24] suggest that ChatGPT provides numerous advantages to a diverse group of users.”
Zero-Shot Information Extraction via Chatting with ChatGPT (2023, methodology): “For example, (Zhang et al., 2022) use ChatGPT achieved state-of-the-art performance on Stance Detection, (Guo et al., 2023) evaluated its helpfulness on question answering, (Jiao et al., 2023) state that it is a good translator for spoken language.”
How is ChatGPT's behavior changing over time? (2023, methodology): “Researchers have evaluated temporal shifts of previous (non-LLM) AI models (L. Chen et al., 2021).”
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “…2023b); it hallu-cinates with human-like fluency and eloquence on things that are not based on truth (Shen et al., 2023; Thorp, 2023; Smith, 2023); and as a general-purpose language model trained from everything on the web, its language coverage is questionable (Lu et al., 2022; Jiao et al., 2023).”
Is ChatGPT a Good Recommender? A Preliminary Study (2023, background): “Jiao et al. [23] finds the translation ability of Chat-GPT performs competitively with commercial translation products on high-resource and low-resource languages.”
Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, background): “For NLG tasks, Jiao et al. (2023) claim that ChatGPT performs competitively with commercial translation products ( e.g. , Google Translator) on high-resource European languages.”
ChatGPT and Open-AI Models: A Preliminary Review (2023, methodology): “Jiao et al. evaluated the machine translation capabilities of ChatGPT in comparison to three commercial translation products: Google Translate, DeepL Translate, and Tencent TranSmart [62].”
ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing (2023, methodology): “Similarly, ChatGPT could be used for translation or revision of imperfect English in manuscripts in order to attain publication in higher quality journals (Jiao et al., 2023).”
Mind the Gap! Choice Independence in Using Multilingual LLMs for Persuasive Co-Writing Tasks in Different Languages (2025, unknown): “Yet, despite their immense promise, LLMs still differ substantially across languages [3, 9, 43, 56, 60, 63, 134].”