ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning
By Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, Thien Huu Nguyen. Conference on Empirical Methods in Natural Language Processing, 2023
415 citations recorded by the source corpus.
Over the last few years, large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) that fundamentally transform research and developments in the field. ChatGPT represents one of the most exciting LLM systems developed recently to showcase impressive skills for language generation and highly attract public attention. Among various exciting applications discovered for ChatGPT in English, the model can process and generate texts for multiple languages due to its multilingual training data. Given the broad adoption of ChatGPT for English in different problems and areas, a natural question is whether ChatGPT can also be applied effectively for other languages or it is necessary to develop more language-specific technologies. The answer to this question requires a thorough evaluation of ChatGPT over multiple tasks with diverse languages and large datasets (i.e., beyond reported anecdotes), which is still missing or limited in current research. Our work aims to fill this gap for the evaluation of ChatGPT and similar LLMs to provide more comprehensive information for multilingual NLP applications. While this work will be an ongoing effort to include additional experiments in the future, our current paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. We also focus on the zero-shot learning setting for ChatGPT to improve reproducibility and better simulate the interactions of general users. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages, calling for further research to develop better models and understanding for multilingual learning.
Subfields: Artificial Intelligence, Health Informatics, Radiology, Nuclear Medicine and Imaging
View Paper: https://www.semanticscholar.org/paper/dfbfa21a93c3164ae8a033398c8de42b03b1b84d
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Health Informatics
In Radiology, Nuclear Medicine and Imaging
Citations in the corpus
- MEGA: Multilingual Evaluation of Generative AI (2023, methodology): “However, we would like to highlight that despite the possibility of contamina-Dataset 2023) also perform multilingual benchmarking of large language models, however, they evaluate the performance of ChatGPT and BLOOMZ in their work while our evaluation also spans GPT-4.”
- A Survey on Evaluation of Large Language Models (2023, neutral): “(Lai et al., 2023) X (Lopez-Lira and Tang, 2023) X (Bian et al.”
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “Nevertheless, ChatGPT still lacks the ability to understand and generate sentences in low-resource languages (§2.2), which is also supported by Lai et al. (2023).”
- SeaLLMs - Large Language Models for Southeast Asia (2023, background): “While the proficiencies of these models are extraordinary, the majority of existing LLMs embody a linguistic hierarchy overwhelmingly dominated by English [1, 13, 37].”
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, neutral): “Some previous studies have investigated the performance of LLMs on some multi-lingual benchmarks (Ahuja et al., 2023; Lai et al., 2023), and collectively found that their performance degenerates when generalizing to non-Latin languages.”
- Low-Resource Languages Jailbreak GPT-4 (2023, result): “More importantly, our results also suggest that researchers have underestimated the capability of the LLMs in understanding and generating text in low-resource languages [4, 18, 22, 39, 46, 48].”