SeaLLMs - Large Language Models for Southeast Asia

By Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu and 2 more. 2023

146 citations recorded by the source corpus.

Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon the Llama-2 model and further advanced through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM-13b models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peiyi Wang
2. Zhihong Shao
3. Runxin Xu
4. Qihao Zhu
5. Y. K. Li
6. Daya Guo
7. Yu Wu
8. Jinyu Li

In Computer Vision and Pattern Recognition

1. Peiyi Wang
2. Zhihong Shao
3. Daya Guo
4. Yu Wu

In Large Language Models

1. Peiyi Wang
2. Zhihong Shao
3. Daya Guo
4. Yu Wu

In Natural Language Processing

1. Peiyi Wang
2. Zhihong Shao
3. Runxin Xu
4. Daya Guo
5. Yu Wu

Citations in the corpus

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “• Open-source models include: general models like (1) DeepSeek-LLM-Chat 67B (DeepSeek-AI, 2024), (2) Qwen 72B (Bai et al., 2023), and (3) ChatGLM3 6B (ChatGLM3 Team, 2023), as well as models with enhancements in mathematics including (4) InternLM2-Math 20B 6 which builds on InternLM2 and underwent…”
• AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages (2026, unknown): “We hope these findings inform more effective adaptation of LLMs to low-resource languages.”