A Nutritional Label for Rankings

By Ke Yang, Julia Stoyanovich, Abolfazl Asudeh, Bill Howe, Hosagrahar Visvesvaraya Jagadish, Gerome Miklau. 2018

116 citations recorded by the source corpus.

Algorithmic decisions often result in scoring and ranking individuals to determine credit worthiness, qualifications for college admissions and employment, and compatibility as dating partners. While automatic and seemingly objective, ranking algorithms can discriminate against individuals and protected groups, and exhibit low diversity. Furthermore, ranked results are often unstable -- small changes in the input data or in the ranking methodology may lead to drastic changes in the output, making the result uninformative and easy to manipulate. Similar concerns apply in cases where items other than individuals are ranked, including colleges, academic departments, or products. Despite the ubiquity of rankers, there is, to the best of our knowledge, no technical work that focuses on making rankers transparent. In this demonstration we present Ranking Facts, a Web-based application that generates a "nutritional label" for rankings. Ranking Facts is made up of a collection of visual widgets that implement our latest research results on fairness, stability, and transparency for rankings, and that communicate details of the ranking methodology, or of the output, to the end user. We will showcase Ranking Facts on real datasets from different domains, including college rankings, criminal risk assessment, and financial services.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “Documenting model performance [117, 128, 90], both overall and within subgroups, is crucial to effective evaluation and can also help check decision systems against some of the “fairness” definitions from Section 4.”
FairSight: Visual Analytics for Fairness in Decision Making (2019, methodology): “With the increasing awareness of the importance of fair machine learning, a number of new tools, including API [3] and interface [40], as well as integrated systems such as the What-if tool [15] or AI Fairness 360 [22], have been developed.”
Fairness and Abstraction in Sociotechnical Systems (2019, background): “In fact, a number of recent papers have proposed ideas along these lines [12, 27, 37, 55, 58, 60, 71].”
The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards (2018, background): ““Ranking Facts” is a web-based system that generates a “nutrition label” for scoring and ranking algorithms based on factors or “widgets” to communicate an algorithm’s methodology or output [24] .”
Explainability fact sheets: a framework for systematic assessment of explainable approaches (2019, methodology): “• quickly grasp properties of a given method to choose an appropriate approach for a desired use case given the unified design of the Fact Sheets (as with food nutrition labels [56] the users know what to expect and how to navigate them).”
Algorithmic Fairness: Choices, Assumptions, and Definitions (2021, background): “Similarly, documenting model performance, both overall and within subgroups, is crucial to effective evaluation and can also help check decision systems against some of the fairness definitions from Section 4 (Mitchell et al. 2018, Stoyanovich 2018, Yang et al. 2018).”
Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science (2018, background): “…“datasheets for datasets,” looking at AI more broadly (but including NLP); Chmielinski and colleagues at the MIT Media Lab propose “dataset nutrition labels”; 19 and Yang et al. (2018) describe “Ranking Facts,” a series of widgets that allow a user to explore how attributes inﬂuence a ranking.”
Toward situated interventions for algorithmic equity: lessons from the field (2020, background): “Other scholars have pursued empirical case studies [19, 50, 52], field evaluations [10, 65], context-aware uses of datasets [46, 63], and questions of institutional access to real-world data [67].”
The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence (2022, background): “Finally, there are several related projects in the dataset transparency space, including the manually-generated Datasheets for Datasets work which our project leverages [9], data statements [30], and several automated tools [31, 23].”