Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model

By Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid and 7 more. Annual Meeting of the Association for Computational Linguistics, 2024

385 citations recorded by the source corpus.

Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages. What does it take to broaden access to breakthroughs beyond first-class citizen languages? Our work introduces Aya, a massively multilingual generative language model that follows instructions in 101 languages of which over 50% are considered as lower-resourced. Aya outperforms mT0 and BLOOMZ on the majority of tasks while covering double the number of languages. We introduce extensive new evaluation suites that broaden the state-of-art for multilingual eval across 99 languages -- including discriminative and generative tasks, human evaluation, and simulated win rates that cover both held-out tasks and in-distribution performance. Furthermore, we conduct detailed investigations on the optimal finetuning mixture composition, data pruning, as well as the toxicity, bias, and safety of our models. We open-source our instruction datasets and our model at https://hf.co/CohereForAI/aya-101

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/2296629527ebbd6f8c897df7cf5cdbac3f0cc15b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, positive): “We specifically focus on English data and evaluations for T ÜLU 3 (although we do include the multilingual Aya [Üstün et al., 2024] dataset due to its high quality).”
UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset (2024, result): “Given that Guanaco-13b shares the same backbone (i.e., Llama-2-13b) with our model, the results imply the superiority of the proposed UltraLink dataset.”
Human or Machine? A Survey on Machine-Generated Text Detection (2026, unknown): “While some languages, such as Arabic and Chinese, have extensive resources for general NLP tasks [33], [34], they remain low-resourced in the MGT detection task.”
Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations (2025, unknown): “…and performance of generative models for everyday uses (Jiang et al., 2023; Grattafiori et al., 2024; Achiam et al., 2023; Anthropic, 2024; ¨Ust¨un et al., 2024; Team et al., 2023) and similar improvements in specialized domains such as medicine (Thirunavukarasu et al., 2023; Nori et al.,…”