AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages
By Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, C. GokulN., Avik Bhattacharyya, Mitesh M. Khapra, Pratyush Kumar. 2020
116 citations recorded by the source corpus.
We present the IndicNLP corpus, a large-scale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. We share pre-trained word embeddings trained on these corpora. We create news article category classification datasets for 9 languages to evaluate the embeddings. We show that the IndicNLP embeddings significantly outperform publicly available pre-trained embedding on multiple evaluation tasks. We hope that the availability of the corpus will accelerate Indic NLP research. The resources are available at this https URL.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Yejin Choi
- Noah A. Smith
- Yonatan Belinkov
- Percy Liang
- Graham Neubig
- Jianfeng Gao
In Natural Language Processing
- Mohit Bansal
- Yonatan Belinkov
- Yejin Choi
- Noah A. Smith
- Sebastian Gehrmann
- Graham Neubig
- Samuel R. Bowman
- Percy Liang
In Representation & Self-Supervised Learning
- Yonatan Belinkov
- Ellie Pavlick
- Niklas Muennighoff
- Timo Schick
- Nouamane Tazi
- Douwe Kiela
- Sebastian Gehrmann
- Anna Rogers
Citations in the corpus
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “…we expanded Swahili in the original selection to the category of Niger-Congo languages, and Hindi and Urdu to 5. hf.co/spaces/bigscience/data_host_provider_agreement 6. allenai.org 7. lemonde.fr 8. hf.co/bigscience-data 9. forms.gle/qyYswbEL5kA23Wu99 Indic languages (Kunchukuttan et al., 2020).”
- Re-contextualizing Fairness in NLP: The Case of India (2022, methodology): “We analyze the Wikipedia corpus used to train LMs like BERT (Devlin et al., 2018b), and the IndicCorp-en corpus used in training multilingual models like IndicBERT (Kakwani et al., 2020).”