Datasets: A Community Library for Natural Language Processing

By Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, A. Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall and 22 more. 2021

775 citations recorded by the source corpus.

The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Beyond identifying issues, standardized documentation following mutually agreed-upon frameworks can lower the barrier of entry to newly developed resources (Lhoest et al., 2021).”
Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face (2023, methodology): “It does so by standing on the shoulders of battle-tested open-source libraries from the Castorini [20] and Hugging Face [1, 19] ecosystems and enhancing the interoperability between them to enable quick indexing and free deployment of search interfaces.”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “In the first phase, we organized an open hackathon to start gathering identified sources on the Hugging Face Datasets hub (Lhoest et al., 2021), in a dedicated organization 5 (in order to manage access controls).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, methodology): “of datasets, like the UCI ML Repository [45] and the Hugging Face Dataset Repository [86], have become popular.”
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources (2022, methodology): “The Hugging Face Datasets library (Lhoest et al., 2021) and Tensorflow library (Tensor Flow Authors, 2021) both provide tools for loading datasets from online as well as locally and include catalogs of directly accessible datasets.”
Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, methodology): “To cover broad and diverse NLP tasks, we select over 100 representative tasks from Huggingface datasets 18 .”
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, neutral): “The Datasets library significantly reduces memory overhead during data processing and accelerates sample querying using Arrow (Apache, 2016).”
MTEB: Massive Text Embedding Benchmark (2022, methodology): “(c) Extensibility: New datasets for existing tasks can be benchmarked in MTEB via a single file that specifies the task and a Hugging Face dataset name where the data has been uploaded (Lhoest et al., 2021).”
Editing Models with Task Arithmetic (2022, neutral): “[53] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas P”