Why Are We Using Black Box Models in AI When We Don’t Need To? A Lesson From An Explainable AI Competition

By Cynthia Rudin, Joanna Radin. 2019

412 citations recorded by the source corpus.

In 2018, a landmark challenge in artificial intelligence (AI) took place, namely, the Explainable Machine Learning Challenge. The goal of the competition was to create a complicated black box model for the dataset and explain how it worked. One team did not follow the rules. Instead of sending in a black box, they created a model that was fully interpretable. This leads to the question of whether the real world of machine learning is similar to the Explainable Machine Learning Challenge, where black box models are used even when they are not needed. We discuss this team’s thought processes during the competition and their implications, which reach far beyond the competition itself.Keywords: interpretability, explainability, machine learning, finance

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “, even the humans who design models do not always understand how variables are being combined to make inferences [171], many respondents discussed an end-to-end opacity of inputs, model behaviour, and outcomes in India.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “, 2019; Lakkaraju and Bastani, 2020), or adding unnecessary authority to the black box (Rudin and Radin, 2019).”
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions (2023, background): “In other words, it becomes increasingly complex to justify the inferences of AI models, both with simpler ante-hoc explainable models [97–99], and even more with complex models [72– 74].”
Human-Centered AI (2021, background): “…[AI and ML] techniques were born and bred for low-stakes decisions such as online advertising and web search where individual decisions do not deeply affect human lives” [18] , AI Engineering in high-consequence contexts necessitates system transparency, predictability, and interpretability.”
Six Human-Centered Artificial Intelligence Grand Challenges (2023, neutral): “Explainability methods can be useful but are also complicated (T. Miller, 2019) and potentially fallible themselves, making them incomplete or inappropriate solutions in some cases (Rudin & Radin, 2019).”