Engineering safety in machine learning

By Kush R. Varshney. 2016

137 citations recorded by the source corpus.

Machine learning algorithms are increasingly influencing our decisions and interacting with us in all parts of our daily lives. Therefore, just like for power plants, highways, and myriad other engineered sociotechnical systems, we must consider the safety of systems involving machine learning. In this paper, we first discuss the definition of safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes. Then we examine dimensions, such as the choice of cost function and the appropriateness of minimizing the empirical average training cost, along which certain real-world applications may not be completely amenable to the foundational principle of modern statistical machine learning: empirical risk minimization. In particular, we note an emerging dichotomy of applications: ones in which safety is important and risk minimization is not the complete story (we name these Type A applications), and ones in which safety is not so critical and risk minimization is sufficient (we name these Type B applications). Finally, we discuss how four different strategies for achieving safety in engineering (inherently safe design, safety reserves, safe fail, and procedural safeguards) can be mapped to the machine learning context through inter-pretability and causality of predictive models, objectives beyond expected prediction accuracy, human involvement for labeling difficult or rare examples, and user experience design of software.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products (2016, background): “To the best of our knowledge, there is no existing work on analyzing machine learning usin g pr cise decision-theoretic definitions of safety except ou r own preliminary work [4].”
Interpretable Two-level Boolean Rule Learning for Classification (2015, methodology): “Due to space constraints, we refer the reader to Malioutov & Varshney(2013) for results from other classifiers that are not interpretable; the accuracy of our algorithms is generally quite competitive with them.”
Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262 (2018, methodology): “Notes The distance to the decision boundary of a classifier has been used to indicate confidence (further means more confident) but this becomes less reliable for rarer inputs [38].”
Artificial intelligence, bias and clinical safety (2019, unknown): “If the system’s confidence is low, best practice design would be to fail-safe 46 and refuse to make a prediction either way.”
On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection (2018, background): “Blindly following machine predictions may lead to dire repercussions, especially in scenarios such as medical diagnosis and justice systems [9, 36, 73].”
An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals (2023, unknown): “By consequence, ensuring high-quality input data as well as ensuring that the system is applied to the anticipated patient population is of utmost importance to AI-based systems ’ acceptance 60 .”
A survey of deep learning techniques for autonomous driving (2019, background): “In Varshney (2016), Varshney defines safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes.”
Software Engineering for AI-Based Systems: A Survey (2021, background): “ML components [117, 200], and specific safety strategies for cyber-physical systems [7, 201].”
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective (2020, background): “Previous work defines safety in machine learning systems as minimizing the possibility and probability of expected harms (Varshney, 2016; Varshney, 2020).”
Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods (2019, background): “…of uncertainty is desirable and should be considered as a key feature of any machine learning method, all the more in safety-critical application domains such as medicine (Yang et al., 2009; Lambrou et al., 2011) or sociotechnical systems (Varshney, 2016; Varshney and Alemzadeh, 2016).”
TIP: Typifying the Interpretability of Procedures (2017, background): “The problem is really only because of estimation on incomplete test sets [51].”
Examples are not enough, learn to criticize! Criticism for Interpretability (2016, background): “As machine learning (ML) methods have become ubiquitous in human decision making, their transparency and interpretability have grown in importance (Varshney, 2016).”
Aleatoric and Epistemic Uncertainty in Machine Learning: A Tutorial Introduction (2019, background): “L G ] 2 application domains such as medicine (Yang et al., 2009; Lambrou et al., 2011) or sociotechnical systems (Varshney, 2016; Varshney and Alemzadeh, 2016).”