iNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages

By Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, Pratyush Kumar. 2020

387 citations recorded by the source corpus.

In this paper, we introduce NLP resources for 11 major Indian languages from two major language families. These resources include: (a) large-scale sentence-level monolingual corpora, (b) pre-trained word embeddings, (c) pre-trained language models, and (d) multiple NLU evaluation datasets (IndicGLUE benchmark). The monolingual corpora contains a total of 8.8 billion tokens across all 11 languages and Indian English, primarily sourced from news crawls. The word embeddings are based on FastText, hence suitable for handling morphological complexity of Indian languages. The pre-trained language models are based on the compact ALBERT model. Lastly, we compile the (IndicGLUE benchmark for Indian language NLU. To this end, we create datasets for the following tasks: Article Genre Classification, Headline Prediction, Wikipedia Section-Title Prediction, Cloze-style Multiple choice QA, Winograd NLI and COPA. We also include publicly available datasets for some Indic languages for tasks like Named Entity Recognition, Cross-lingual Sentence Retrieval, Paraphrase detection, etc. Our embeddings are competitive or better than existing pre-trained embeddings on multiple tasks. We hope that the availability of the dataset will accelerate Indic NLP research which has the potential to impact more than a billion people. It can also help the community in evaluating advances in NLP over a more diverse pool of languages. The data and models are available at https://indicnlp.ai4bharat.org.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Re-contextualizing Fairness in NLP: The Case of India (2022, methodology): “We analyze the Wikipedia corpus used to train LMs like BERT (Devlin et al., 2018b), and the IndicCorp-en corpus used in training multilingual models like IndicBERT (Kakwani et al., 2020).”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, methodology): “We evaluate the models on the dev set of X-CSQA with 1,000 samples for each language, while the Wiki Cloze QA dataset from IndicNLPSuite contains 62,314 samples for all languages.”
A Primer on Pretrained Multilingual Language Models (2021, neutral): “Some MLLMs like XLM-R are massively multilingual as they support ∼ 100 languages whereas others like IndicBERT (Kakwani et al., 2020) and MuRIL (Khanuja et al., 2021a) support a smaller set of languages.”
mGPT: Few-Shot Learners Go Multilingual (2022, background): “BERT (Devlin et al., 2019) has been replicated in other high-resource languages (Martin et al., 2020; Masala et al., 2020) and language families, e.g., Indian (Kakwani et al., 2020) and Balto-Slavic (Arkhipov et al., 2019).”