Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus

By Jack Bandy, Jack Bandy, Nicholas Vincent. 2021

80 citations recorded by the source corpus.

Recent literature has underscored the importance of dataset documentation work for machine learning, and part of this work involves addressing"documentation debt"for datasets that have been used widely but documented sparsely. This paper aims to help address documentation debt for BookCorpus, a popular text dataset for training large language models. Notably, researchers have used BookCorpus to train OpenAI's GPT-N models and Google's BERT models, even though little to no documentation exists about the dataset's motivation, composition, collection process, etc. We offer a preliminary datasheet that provides key context and information about BookCorpus, highlighting several notable deficiencies. In particular, we find evidence that (1) BookCorpus likely violates copyright restrictions for many books, (2) BookCorpus contains thousands of duplicated books, and (3) BookCorpus exhibits significant skews in genre representation. We also find hints of other potential deficiencies that call for future research, including problematic content, potential skews in religious representation, and lopsided author contributions. While more work remains, this initial effort to provide a datasheet for BookCorpus adds to growing literature that urges more careful and systematic documentation for machine learning datasets.

Subfields: Artificial Intelligence, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

Deduplicating Training Data Makes Language Models Better (2021, methodology): “The Book Corpus (Zhu et al., 2015), which was used to train popular models such as BERT, has a substantial amount of exact-duplicate documents according to Bandy and Vincent (2021).”
The ROOTS Search Tool: Data Transparency for LLMs (2023, methodology): “…to the permissions to use the data, which, perhaps the most famously, surfaced in relation to the BookCor-pus (Zhu et al., 2015), used, among others, to train BERT (Devlin et al., 2019), but collected without author permissions and eventually taken down by the authors (Bandy and Vincent, 2021).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “Other LM pre-training datasets that have been documented and analyzed include C4 (Dodge et al., 2021; Luccioni and Viviano, 2021; Kreutzer et al., 2022), OSCAR (Kreutzer et al., 2022) and BookCorpus (Bandy and Vincent, 2021) .”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
The MiniPile Challenge for Data-Efficient Language Models (2023, background): “Bookcorpus [4] contains >7k unpublished books with long stretches of contiguous text.”
Changing the World by Changing the Data (2021, background): “The tide seems to be turning in this direction: since this work was submitted for review, at least two papers came out documenting popular resources for pre-training language models (Dodge et al., 2021; Bandy and Vincent, 2021).”
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication (2021, background): “Addressing ‘documentation debt’ [6] and creating datasheets [34] to promote transparency can spur dataset users to make better informed choices about the relevance of a dataset, even if it has not been deprecated.”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “Nevertheless, many quality issues persist, as discussed by Bandy and Vincent [9] and Dodge et al. [32].”
Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP (2021, unknown): “trained language models, there has arguably been an over-focus on data volume and too little focus on any of the data principles, including the use of corpora of dubious legality and quality such as BookCorpus (Bandy and Vincent, 2021).”
Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? (2022, unknown): “Also, in the process of creating a datasheet for the BookCorpus, Bandy and Vincent [2021] stated that further research is necessary to explore the Fig.”
Frequency Effects on Syntactic Rule Learning in Transformers (2021, methodology): “However, BooksCorpus is not publicly available (Bandy and Vincent, 2021), so when we pre-train our BERT models, we use only the Wikipedia data (2.3 billion tokens).”