Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards
By Angelina McMillan-Major, Salomey Osei, Juan Diego Rodriguez, Pawan Sasanka Ammanamanchi, Sebastian Gehrmann, Yacine Jernite. 2021
58 citations recorded by the source corpus.
Developing documentation guidelines and easy-to-use templates for datasets and models is a challenging task, especially given the variety of backgrounds, skills, and incentives of the people involved in the building of natural language processing (NLP) tools. Nevertheless, the adoption of standard documentation practices across the field of NLP promotes more accessible and detailed descriptions of NLP datasets and models, while supporting researchers and developers in reflecting on their work. To help with the standardization of documentation, we present two case studies of efforts that aim to develop reusable documentation templates – the HuggingFace data card, a general purpose card for datasets in NLP, and the GEM benchmark data and model cards with a focus on natural language generation. We describe our process for developing these templates, including the identification of relevant stakeholder groups, the definition of a set of guiding principles, the use of existing templates as our foundation, and iterative revisions based on feedback.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Mohit Bansal
- Percy Liang
- Sebastian Gehrmann
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Noah A. Smith
In Information Systems
- Percy Liang
- Ondrej Dusek
- Sebastian Gehrmann
- Leonardo F. R. Ribeiro
- Vitaly Nikolaev
- Thibault Sellam
- Elizabeth Clark
- Jekaterina Novikova
In Natural Language Processing
- Sebastian Gehrmann
- Percy Liang
- Yejin Choi
- Elizabeth Clark
- Mohit Bansal
- Thibault Sellam
- Jianfeng Gao
- Ondrej Dusek
In Evaluation & Benchmarks
- Elizabeth Clark
- Sebastian Gehrmann
- Thibault Sellam
- Percy Liang
- Ondrej Dusek
- Alexander M. Rush
- Jekaterina Novikova
- Michael. White
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…the choices of dataset creators, it is imperative to report the curation decisions alongside limitations of datasets in structured format to allow for a better interpretation and contextualization of performance results (Bender and Friedman, 2018; Gebru et al., 2021; McMillan-Major et al., 2021).”
- Datasets: A Community Library for Natural Language Processing (2021, background): “…a set of structured tags holding information about their languages, tasks supported, licenses, etc.; a data card based on a template 5 designed to combine relevant technical considerations and broader context information (McMillan-Major et al., 2021); and a list of models trained on the dataset.”
- Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “Public repositories do present a unique opportunity to help harmonize emerging standards around documentation [91] mentioned in Section 3.”
- GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022, methodology): “Our original data documentation process (McMillan-Major et al., 2021) required filling out a markdown template following instructions in a separate guide.”