BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model
By Christopher Akiki, Giada Pistilli, Margot Mieskes, Matthias Gallé, Thomas Wolf, Suzana Ili'c, Yacine Jernite. 2022
15 citations recorded by the source corpus.
The BigScience Workshop was a value-driven initiative that spanned one and half years of interdisciplinary research and culminated in the creation of ROOTS, a 1.6TB multilingual dataset that was used to train BLOOM, one of the largest multilingual language models to date. In addition to the technical outcomes and artifacts, the workshop fostered multidisciplinary collaborations around large models, datasets, and their analysis. This in turn led to a wide range of research publications spanning topics from ethics to law, data governance, modeling choices and distributed training. This paper focuses on the collaborative research aspects of BigScience and takes a step back to look at the challenges of large-scale participatory research, with respect to participant diversity and the tasks required to successfully carry out such a project. Our main goal is to share the lessons we learned from this experience, what we could have done better and what we did well. We show how the impact of such a social approach to scientific research goes well beyond the technical artifacts that were the basis of its inception.
Subfields: General Social Sciences, Information Systems and Management, Safety Research, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In General Social Sciences
- Yonatan Belinkov
- Yada Pruksachatkun
- Stella Biderman
- Jungo Kasai
- Aleksandra Piktus
- Teven Le Scao
- Angela Fan
- Javier de la Rosa
In Information Systems and Management
- Irene Solaiman
- Jimmy J. Lin
- Mohammad Shoeybi
- D. Narayanan
- Benjamin Heinzerling
- Kyle Lo
- Margaret Mitchell
- Canwen Xu
In Safety Research
- Irene Solaiman
- Pascale Fung
- Sebastian Gehrmann
- Stella Biderman
- Anna Rogers
- Pawan Sasanka Ammanamanchi
- Salomey Osei
- Tristan Thrush
In Large Language Models
- Timo Schick
- Colin Raffel
- Pascale Fung
- Luke S. Zettlemoyer
- Jimmy J. Lin
- Douwe Kiela
- Alexander M. Rush
- Niklas Muennighoff
Citations in the corpus
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “In particular, the charter emphasizes values of inclusivity and diversity, openness and reproducibility, and responsibility in various aspects of the organization (Akiki et al., 2022).”
- StarCoder: may the source be with you! (2023, neutral): “One example is the BigScience research workshop (BigScience Workshop, 2022), an open scientific collaboration (Akiki et al., 2022) comprising hundreds of researchers collaborating to release BLOOM, a multi-lingual LLM (Scao et al., 2022; Muennigho � et al., 2022).”
- Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face (2023, background): “This often leads to a “train first, ask questions later” approach to model training [2], itself a concrete instance of convenience experimentation [16], an approach to research that is modulated by the availability of a resource and ease of a method, rather than the suitability thereof to the proble”
- The ROOTS Search Tool: Data Transparency for LLMs (2023, background): “This raises questions concerning the quality of the data, the feasibility of curating and in-specting it, as well as documenting it in terms of what kinds of speech and speakers it represents (Jo and Gebru, 2020; Bender et al., 2021; Akiki et al., 2022).”