On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products

By Kush R. Varshney, Homa Alemzadeh. 2016

252 citations recorded by the source corpus.

Abstract Machine learning algorithms increasingly influence our decisions and interact with us in all parts of our daily lives. Therefore, just as we consider the safety of power plants, highways, and a variety of other engineered socio-technical systems, we must also take into account the safety of systems involving machine learning. Heretofore, the definition of safety has not been formalized in a machine learning context. In this article, we do so by defining machine learning safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes. We then use this definition to examine safety in all sorts of applications in cyber-physical systems, decision sciences, and data products. We find that the foundational principle of modern statistical machine learning, empirical risk minimization, is not always a sufficient objective. We discuss how four different categories of strategies for achieving safety in engineering, including inherently safe design, safety reserves, safe fail, and procedural safeguards can be mapped to a machine learning context. We then discuss example techniques that can be adopted in each category, such as considering interpretability and causality of predictive models, objective functions beyond expected prediction accuracy, human involvement for labeling difficult or rare examples, and user experience design of software and open data.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Robotics & Control

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, neutral): “…been studied in regards to processes that depend on the output of AI models, such as vehicular perception and self-driving in autonomous vehicles, automated surgery, data-based support for medical diagnosis, insurance risk assessment and cyber-physical systems in manufacturing, among others [369] .”
Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (2018, background): “models stating that dangerous situations are safe [McGough, 2018] and generally poor use of limited valuable resources in criminal justice, medicine, energy reliability, ﬁnance, and in other domains [Varshney and Alemzadeh, 2016]. Rather than trying to create models that are inherently interpretable”
Please Stop Explaining Black Box Models for High Stakes Decisions (2018, background): “The lack of transparency and accountability of predictive models can have (and has had already) severe consequences; there have been cases of people incorrectly denied parole [Wexler, 2017], poor bail decisions leading to the release of dangerous criminals, ML-based pollution models stating that dan”
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “38 As an operation continues, models may be retrained on more relevant data, but the amount of data available will be limited (and possibly conﬂicting and of low quality).”
Increasing Trust in AI Services through Supplier's Declarations of Conformity (2018, background): “…interpretation) [38], in which a person can look at a model and understand what it does, reduces epistemic uncertainty and increases safety because quirks and vagaries of training dataset distributions that will not be present in distributions during deployment can be identiﬁed by inspection [2].”
To Trust Or Not To Trust A Classifier (2018, background): “As such, ML trust and safety is an important theme [1, 2, 3].”
Probabilistic Object Detection: Definition and Evaluation (2018, background): “For safe and trusted operation in robots or autonomous systems, CNNs must express meaningful uncertainty information [1, 14, 15, 19, 43, 49].”
Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer (2017, background): “Varshney and Alemzadeh [32] propose “safety reserves” and “safe fail” options which combine learning with rejection and fairness/safety, but do not analyze learning procedures or larger decision-making frameworks.”
FactSheets: Increasing Trust in AI Services through Supplier's Declarations of Conformity. (2018, background): “[3] K.”
Improving Confidence Estimates for Unfamiliar Examples (2018, background): “Scholars that study the impact of AI on society consider differently distributed samples to be a major risk [49]: “This is one form of epistemic uncertainty that is quite relevant to safety because training on a dataset from a different distribution can cause much harm.””
Bias Mitigation Post-processing for Individual and Group Fairness (2018, background): “Without bias mitigation, models trained on such decisions perpetuate and scale human biases and are thereby unsafe and untrustworthy [3, 4].”
Towards A Rigorous Science of Interpretable Machine Learning (2017, background): “…in systems optimized not only for expected task performance but also other important criteria such as safety [Otte, 2013, Amodei et al., 2016, Varshney and Alemzadeh, 2016], nondiscrimination [Bostrom and Yudkowsky, 2014, Ruggieri et al., 2010, Hardt et al., 2016], avoiding technical debt…”
A survey of deep learning techniques for autonomous driving (2019, background): “This uncertainty caused by the instantiation of the test set can have large safety implications on individual test samples [136].”
Software Engineering for AI-Based Systems: A Survey (2021, background): “We can highlight that in the 11 primary studies discussing roadmaps, they are often related to safety and the standard ISO 26262 [7, 66, 72, 105, 201].”