FactSheets: Increasing trust in AI services through supplier's declarations of conformity

By Matthew Arnold, Rachel K. E. Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilovic, Ravi Nair, Karthikeyan Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski and 4 more. 2019

370 citations recorded by the source corpus.

Accuracy is an important concern for suppliers of artificial intelligence (AI) services, but considerations beyond accuracy, such as safety (which includes fairness and explainability), security, and provenance, are also critical elements to engender consumers’ trust in a service. Many industries use transparent, standardized, but often not legally required documents called supplier's declarations of conformity (SDoCs) to describe the lineage of a product along with the safety and performance testing it has undergone. SDoCs may be considered multidimensional fact sheets that capture and quantify various aspects of the product and its development to make it worthy of consumers’ trust. In this article, inspired by this practice, we propose FactSheets to help increase trust in AI services. We envision such documents to contain purpose, performance, safety, security, and provenance information to be completed by AI service providers for examination by consumers. We suggest a comprehensive set of declaration items tailored to AI in the Appendix of this article.

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “• Finally, the README asks users to develop appropriate documentation (PAI, 2019; Arnold et al., 2018; Mitchell et al., 2019) when building on CTRL and to tell the research team how they are using CTRL by emailing ctrl-monitoring@salesforce.com .”
Behavioral Use Licensing for Responsible AI (2020, methodology): “We then describe exemplar clauses based on some ethical AI principles and information shared in AI Factsheets [11].”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “The requirements for model documentation or “About Me” page, which provides not only an overview of the model characteristics but also how it is developed and intended to be used, are discussed in recent literature as critical to AI transparency and governance [9, 102].”
Interpretable Machine Learning (2019, background): “Researchers have proposed the idea of fact sheets for ML systems and their underlying data.(111) A fact sheet could give”
Pretrained AI Models: Performativity, Mobility, and Change (2019, background): “Pretrained models are mathematical objects that specify particular neural network architectures and learned synaptic weights: they are functional and can be used directly to perform inference when deployed as AI services or as part of larger AI services in sociotechnical systems, cf. [3, 90].”
FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare (2023, neutral): “This will increase transparency and accountability by providing detailed and continuous information on the AI tools during their lifetime to clinicians, healthcare organisations, citizens and patients, AI developers, and relevant authorities.”
The ethics of algorithms: key problems and solutions (2020, neutral): “Unfortunately, publicly available documentation is currently uncommon in the development of algorithmic systems and there is no agreed-upon format for what should be included when documenting the origin of a dataset (Arnold et al. 2019; Gebru et al. 2020).”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, methodology): “Standardized documentation frameworks have been proposed to encourage both reflection and transparency around models [38, 143], AI services [6], and training and evaluation datasets [15, 58, 76].”
Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards (2021, background): “Arnold et al. (2019) introduce Fact-Sheets to document the use, performance and security aspects of an AI product.”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, methodology): “• Show how the prediction changes corresponding to the inquired change of input PDP [49], ALE [10], ICE [44] Performance See examples in FactSheets [11] and Datasheets [39] Output • Describe the scope of output or system functions.”
Trust in AI: progress, challenges, and future directions (2024, neutral): “Several technical aspects of trust are proposed to be addressed in AI systems, including reliability, safety (encompassing fairness and explainability), security, and lineage (Bore et al., 2018); (Arnold, Bellamy et al., 2019a).”
Explainability fact sheets: a framework for systematic assessment of explainable approaches (2019, background): “[1] suggested “fact sheets” for ML services to communicate their capabilities, constrains, biases and transparency.”
Overcoming Failures of Imagination in AI Infused System Development and Deployment (2020, background): “Some of these calls have been realized through structured documentation and checklists aiming at codifying a range of responsible AI principles (Arnold et al., 2019; Mitchell et al., 2019; Gebru et al., 2018; Sokol and Flach, 2020; Stoyanovich and Howe, 2019; Madaio et al., 2020; Kiran et al.,…”
The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence (2022, background): “Several interventions have been researched and deployed at the model level, both within industry [20, 1] and at the government level [4].”