Model Cards for Model Reporting

By Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. 2018

2,935 citations recorded by the source corpus.

Trained machine learning models are increasingly used to perform high-impact tasks in areas such as law enforcement, medicine, education, and employment. In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics. In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information. While we focus primarily on human-centered machine learning models in the application fields of computer vision and natural language processing, this framework can be used to document any trained machine learning model. To solidify the concept, we provide cards for two supervised models: One trained to detect smiling faces in images, and one trained to detect toxic comments in text. We propose model cards as a step towards the responsible democratization of machine learning and related artificial intelligence technology, increasing transparency into how well artificial intelligence technology works. We hope this work encourages those releasing trained machine learning models to accompany model releases with similar detailed evaluation numbers and other relevant documentation.

Subfields: Artificial Intelligence, Political Science and International Relations, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Percy Liang
  4. Jianfeng Gao
  5. Yejin Choi
  6. Quoc V. Le
  7. Noah A. Smith
  8. Yoshua Bengio

In Political Science and International Relations

  1. Maarten Sap
  2. Yejin Choi
  3. Noah A. Smith
  4. Irene Solaiman
  5. Daniel S. Schiff
  6. Eric Michael Smith
  7. Adina Williams
  8. Christopher D. Manning

In Safety Research

  1. Emilio Ferrara
  2. Emily L. Denton
  3. Daniel S. Schiff
  4. A. Hanna
  5. Q. Liao
  6. Ben Hutchinson
  7. Irene Solaiman
  8. Inioluwa Deborah Raji

In Safety, Robustness & Interpretability

  1. Adina Williams
  2. Yulia Tsvetkov
  3. Eric Michael Smith
  4. Yejin Choi
  5. Kai-Wei Chang
  6. Dirk Hovy
  7. Noah A. Smith
  8. Percy Liang

Citations in the corpus