The State and Fate of Linguistic Diversity and Inclusion in the NLP World

By Pratik M. Joshi, Sebastin Santy, A. Budhiraja, Kalika Bali, Monojit Choudhury. 2020

1,384 citations recorded by the source corpus.

Language technologies contribute to promoting multilingualism and linguistic diversity around the world. However, only a very small number of the over 7000 languages of the world are represented in the rapidly evolving language technologies and applications. In this paper we look at the relation between the types of languages, resources, and their representation in NLP conferences to understand the trajectory that different languages have followed over time. Our quantitative investigation underlines the disparity between languages, especially in terms of their resources, and calls into question the “language agnostic” status of current models and systems. Through this paper, we attempt to convince the ACL community to prioritise the resolution of the predicaments highlighted here, so that no language is left behind.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “(See Joshi et al. (2020) for a discussion.)”
Challenges in Detoxifying Language Models (2021, background): “throughout the world (Joshi et al., 2020), and we recommend caution when generalizing our findings to non-English LMs.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Joshi et al. (2020) assess the language diversity in NLP, showing that very few languages beyond English are being studied, regardless of the number of their speakers.”
Learning Compact Metrics for MT (2021, neutral): “, 2020), as well as increasing linguistic coverage (Joshi et al., 2020).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “This means that work on languages with little available data can potentially impact many more people than work on highly resourced languages (Joshi et al., 2020).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Models may fail to comprehend or generate “non-standard” dialects and sociolects, essentially excluding their speakers from their user base (Joshi et al., 2020; Koenecke et al., 2020; Winata et al., 2021).”
Alignment of Language Agents (2021, background): “• Uneven performance for certain speaker groups, of certain languages and dialects (Joshi et al., 2020).”
BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model (2022, background): “While improving the representation of non-European languages in NLP technology can be a worthy goal (Joshi et al., 2020), attempts to develop resources under the full direction and ownership of a handful of institutions outside of their context become extractive “helicopter research” (Haelewaters…”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “Despite the need for evaluating LLMs for a wide range of languages, bias benchmarks that cover non-English languages are rare (Zhou et al., 2019; Joshi et al., 2020).”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “…of the world (Asia, in particular) do have some numerical representation among ACL authors, attendees, and fellows, NLP research over-whelmingly favors a small set of languages, with a heavy skew towards European languages (Joshi et al., 2020) and ‘standard’ language varieties (Kumar et al., 2021).”
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning (2020, background): “Since resource-rich languages tend to belong to only a few families and areas, their sample is highly biased and is not indicative of the expected model performance (Gerz et al., 2018; Ponti et al., 2019a; Joshi et al., 2020).”
Taxonomy of Risks posed by Language Models (2022, background): “training datasets, such as Javanese which is spoken by more than 80 million people [95].”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, neutral): “As our work shows that 100 million tokens are needed for effective adaptation to prompt in a new language (see Section 4.4), a truly low-resource language usually lacks sufficient unlabeled data for such adaptation (Joshi et al., 2020).”
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2021, methodology): “Joshi et al. (2020b) categorized Bangla in the language group that lacks efforts in labeled data collection and relies on self-supervised pretraining (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019) to boost the natural language understanding (NLU) task performances.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, methodology): “Either way, these models do not address the inclusion problems raised by [65], who note that over 90% of”
MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021, background): “In contrast, MAD-G’s multilingual MLM training allows the generation of task-agnostic LAs that can support downstream cross-lingual transfer for arbitrary NLP tasks.”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, background): “Neglecting typological diversity inevitably leads to poor generalizability and the induction of biases (Gerz et al., 2018; Joshi et al., 2020; Ponti et al., 2019).”