MEGA: Multilingual Evaluation of Generative AI

By Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Krithika Ramesh, S. Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed and 2 more. Conference on Empirical Methods in Natural Language Processing, 2023

418 citations recorded by the source corpus.

Generative AI models have shown impressive performance on many Natural Language Processing tasks such as language understanding, reasoning, and language generation. An important question being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative LLMs have been restricted to English and it is unclear how capable these models are at understanding and generating text in other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages. We compare the performance of generative LLMs including Chat-GPT and GPT-4 to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and tasks and discuss challenges in improving the performance of generative LLMs on low-resource languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.

View Paper: https://www.semanticscholar.org/paper/62ad7ea9467bbcdbfe325b9ee561cab3908e4583

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, methodology): “We evaluate gender bias that occurs in translations of different languages (Ahuja et al., 2023) linked in a pro-stereotypical and anti-stereotypical manner.”
A Survey on Evaluation of Large Language Models (2023, methodology): “Despite translating the input to English and using it as the query, generative LLMs still displays subpar performance across tasks and languages compared to SOTA models [2].”
SeaLLMs - Large Language Models for Southeast Asia (2023, background): “While the proficiencies of these models are extraordinary, the majority of existing LLMs embody a linguistic hierarchy overwhelmingly dominated by English [1, 13, 37].”
Do Llamas Work in English? On the Latent Language of Multilingual Transformers (2024, methodology): “It might thus be tempting to conclude that, yes, Llama-2 uses English as an implicit pivot, similar to researchers’ prior use of English as an explicit pivot (Shi et al., 2022; Ahuja et al., 2023; Huang et al., 2023).”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “Some previous studies have investigated the performance of LLMs on some multi-lingual benchmarks (Ahuja et al., 2023; Lai et al., 2023), and collectively found that their performance degenerates when generalizing to non-Latin languages.”
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, unknown): “…including subject knowledge (Clark et al., 2018; Hendrycks et al., 2020), safety (Zhang et al., 2023b; Seth et al., 2023), multilingual proficiency (Ahuja et al., 2023), multidisciplinary understanding (Yue et al., 2024; Hu et al., 2024), perception tests (Yuan et al., 2023), mathematical…”
Generative artificial intelligence: a systematic review and applications (2024, unknown): “Table 7 Performance comparison among different models on all languages within the XNLI dataset, Source:[2].”