IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

By Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar and 1 more. AACL, 2020

491 citations recorded by the source corpus.

Although Indonesian is known to be the fourth most frequently used language over the internet, the research progress on this language in natural language processing (NLP) is slow-moving due to a lack of available resources. In response, we introduce the first-ever vast resource for training, evaluation, and benchmarking on Indonesian natural language understanding (IndoNLU) tasks. IndoNLU includes twelve tasks, ranging from single sentence classification to pair-sentences sequence labeling with different levels of complexity. The datasets for the tasks lie in different domains and styles to ensure task diversity. We also provide a set of Indonesian pre-trained models (IndoBERT) trained from a large and clean Indonesian dataset (Indo4B) collected from publicly available sources such as social media texts, blogs, news, and websites. We release baseline models for all twelve tasks, as well as the framework for benchmark evaluation, thus enabling everyone to benchmark their system performances.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/26da56580fcf8fc6dd5644ede6575bdbbd7caf1a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “Notably, due to the challenges in collecting few-shot data in low-resource languages (i.e. Swahili and Tamil), we only provide few-shot samples for three languages (i.e. Indonesian, Mandarin and Turkish).”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, background): “Other non-English benchmarks focus on specific linguistic phenomena, e.g. code-switching (Khanuja et al., 2020); languages, e.g. Indonesian (Willie et al., 2020) and Persian (Khashabi et al., 2020); and language families, e.g. Indian languages (Kakwani et al., 2020).”
IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP (2020, methodology): “3 Turns out we weren’t the ﬁrst to think to train a monolingual BERT model for Indonesian, or to name it IndoBERT, with (at least) two contemporaneous BERT models for Indonesian that are named “IndoBERT”: Azhari and Lintang (2020) and Wilie et al. (2020).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “T RANSFORMER-based pretrained language models(T-PTLMs) like GPT-1 [1], BERT [2], XLNet [3], RoBERTa [4], ELECTRA [5], T5 [6], ALBERT [7], BART [8] and PEGAUSUS [9] have achieved tremendous success in NLP because of their ability to learn universal language representations from large volumes of unlab”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “The ID IndoBERT (Wilie et al., 2020) model (indobenchmark/indobert-base-p2), the RU RuBERT (Kuratov and Arkhipov, 2019) model (DeepPavlov/rubert-base-cased) by DeepPavlov , and the ZH Chinese-BERT (Devlin et al., 2019) model (bert-base-chinese) by Google are, to the best of our knowledge, the only…”
GPT is an effective tool for multilingual psychological text analysis (2024, unknown): “This was a dataset of tweets labeled with one of five emotions—anger, joy, sadness, fear, and love—by two annotators, taken from a previous study (86).”
Language Models are Few-shot Multilingual Learners (2021, methodology): “…are to pre-train models with self-supervised learning using un-labelled monolingual text data collected from various resources available online (Wilie et al., 2020; Le et al., 2020; Martin et al., 2020; Eddine et al., 2020; Nguyen and Nguyen, 2020; Scheible et al., 2020; Bhattacharjee et al.,…”