Can we open the black box of AI?

By Davide Castelvecchi. 2016

1,646 citations recorded by the source corpus.

Artificial intelligence is everywhere. But before scientists trust it, they first need to understand how machines learn.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “Not being able to understand what has been captured by the model [4] and stored in its internal representation may entail a privacy breach.”
An overview of deep learning in medical imaging focusing on MRI (2018, background): “This is the “black box” problem of deep neural networks [344].”
Biological Sequence Modeling with Convolutional Kernel Networks (2017, neutral): “An important one is their lack of interpretability [9].”
Introducing Artificial Intelligence Training in Medical Education (2019, positive): “Along with changing the conventional ways physician work, the black box problem [29] and liability issues [30] are some of the most anticipated challenges.”
Calibration of Pre-trained Transformers (2020, background): “Neural networks have seen wide adoption but are frequently criticized for being black boxes, offering little insight as to why predictions are made (Benı́tez et al., 1997; Dayhoff and DeLeo, 2001; Castelvecchi, 2016) and making it difficult to diagnose errors at test-time.”
From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment. (2023, background): “, from1 to10) shouldbecarefully designed and calibrated to user expectations.(128)”
A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods (2022, background): “to comprehend the inner workings of learned models [18, 20].”
Generating Counterfactual and Contrastive Explanations using SHAP (2019, background): “AI often has a critique that when effective, it is a black box [Castelvecchi, 2016].”
User Perceptions and Trust of Explainable Machine Learning Fake News Detectors (2022, background): “The Context: Explainable ML Fake News Detectors The lack of transparency in algorithm processes had led to calls for research on explainability in AI (Castelvecchi, 2016).”
Artificial Intelligence and the Public Sector—Applications and Challenges (2018, background): “In this connection, Castelvecchi (2016, p. 21) highlights the opaqueness of AI systems and notes that “ [i]nstead of storing what they have learned in a neat block of digital memory, they diffuse the information in a way that is exceedingly difficult to decipher, ” which in turn makes them also…”
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey (2020, background): “However, modern API-based services produce more challenges because of the relative ‘black-box’ [42] nature of the problem where the end-user have information only about the input provided to the deep learning model and not the model itself.”
Artificial Intelligence and Entrepreneurship: Implications for Venture Creation in the Fourth Industrial Revolution (2020, background): “A further danger for rapidly scaling companies can be found in the lack of transparency over how some deep learning systems arrive at their outputs (Castelvecchi, 2016).”
Your evidence? Machine learning algorithms for medical diagnosis and prediction (2019, background): “The reason for this above mentioned predicament is that most current machine learning algorithms give rise to what has been called “ the black-box problem ” (Castelvecchi, 2016).”
Interpretable Machine Learning (2019, background): “Some types of ML, such as deep learning (Box 1) are very complex, meaning it may be difficult or impossible to fully understand how a decision has been reached.(28,29) These systems are commonly referred to as ‘black box’ ML.”