Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources

By Angelina McMillan-Major, Zaid Alyafeai, Stella Biderman, Kimbo Chen, F. Toni, Gérard Dupont, Hady ElSahar, Chris C. Emezue, Alham Fikri Aji, Suzana Ili'c and 8 more. 2022

14 citations recorded by the source corpus.

In recent years, large-scale data collection efforts have prioritized the amount of data collected in order to improve the modeling capabilities of large language models. This prioritization, however, has resulted in concerns with respect to the rights of data subjects represented in data collections, particularly when considering the difficulty in interrogating these collections due to insufficient documentation and tools for analysis. Mindful of these pitfalls, we present our methodology for a documentation-first, human-centered data collection project as part of the BigScience initiative. We identified a geographically diverse set of target language groups (Arabic, Basque, Chinese, Catalan, English, French, Indic languages, Indonesian, Niger-Congo languages, Portuguese, Spanish, and Vietnamese, as well as programming languages) for which to collect metadata on potential data sources. To structure this effort, we developed our online catalogue as a supporting tool for gathering metadata through organized public hackathons. We present our development process; analyses of the resulting resource metadata, including distributions over languages, regions, and resource types; and our lessons learned in this endeavor.

Subfields: Communication, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Communication

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “This took the form of hackathons that were co-organized by communities such as Machine Learning Tokyo, Masakhane, and LatinX in AI (McMillan-Major et al., 2022).”
BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model (2022, methodology): “It also informed how we ran e.g. data sourcing hackathons (McMillan-Major et al., 2022) where participants were asked to index language resources that were of broad interest to their work not restricted by their ﬁtness to our speciﬁc use case to make the resulting catalog useful beyond BigScience.”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “The first such effort consisted in creating a tool to support metadata collection through open submissions, called the BigScience Catalogue and running a series of hackathons in collaboration with locally-focused ML and NLP communities such as Masakhane, Machine Learning Tokyo and LatinX in AI where”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “[90] Angelina McMillan-Major, Zaid Alyafeai, Stella Biderman, Kimbo Chen, Francesco De Toni, Gerard Dupont, Hady Elsahar, Chris Emezue, Alham Fikri Aji, Suzana Ilic, Nurulaqilla Khamis, Colin Leong, Maraim Masoud, Aitor Soroa, Pedro Ortiz Suarez, Zeerak Talat, Daniel van Strien, and Yacine Jernite.”
Emergent and Predictable Memorization in Large Language Models (2023, methodology): “Additionally, we were able to obtain access to the ROOTS dataset [McMillan-Major et al., 2022, Laurenc¸on et al., 2022] that the BigScience Workshop’s BLOOM [Scao et al., 2022a] model was trained on.”