Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure

By Besmira Nushi, Ece Kamar, Eric Horvitz. 2018

155 citations recorded by the source corpus.

As machine learning systems move from computer-science laboratories into the open world, their accountability becomes a high priority problem. Accountability requires deep understanding of system behavior and its failures. Current evaluation methods such as single-score error metrics and confusion matrices provide aggregate views of system performance that hide important shortcomings. Understanding details about failures is important for identifying pathways for refinement, communicating the reliability of systems in different settings, and for specifying appropriate human oversight and engagement. Characterization of failures and shortcomings is particularly complex for systems composed of multiple machine learned components. For such systems, existing evaluation methods have limited expressiveness in describing and explaining the relationship among input content, the internal states of system components, and final output quality. We present Pandora, a set of hybrid human-machine methods and tools for describing and explaining system failures. Pandora leverages both human and system-generated observations to summarize conditions of system malfunction with respect to the input content and system architecture. We share results of a case study with a machine learning pipeline for image captioning that show how detailed performance views can be beneficial for analysis and debugging.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, background): “high-dimensional performance analyses to understand the effects of demographic factors, sociocultural factors, and environmental factors on face-based AI systems [51, 61].”
Software Engineering for AI-Based Systems: A Survey (2021, background): “Two papers addressed testing-related explanations for ML systems [112, 148].”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “…to the benchmarking paradigm, such as test suites [1, 126, 170], for example, can re-orient ML evaluation away from its current focus on the competitive determination of “state of the art”, and more towards an exploratory and descriptive probing of model capabilities [65, 106, 143, 161, 169].”
A Survey of Human-in-the-loop for Machine Learning (2021, background): “Since these metrics can only evaluate the predictive performance, they lack the mechanisms to explain when and why the sentiment models give false predictions in run-time [106].”
An Empirical Analysis of Backward Compatibility in Machine Learning Systems (2020, background): “New performance evaluation [7, 31, 41] approaches suggest reporting performance on data slices rather than aggregate measures that overlook partial model regression.”
The role of artificial intelligence in achieving the Sustainable Development Goals (2019, background): “In this sense, research aimed at obtaining insight on the reasons for failure of AI systems, introducing combined human – machine analysis tools 48 , are an essential step towards accountable AI technology, given the large risk associated to such a failure.”
HARK Side of Deep Learning - From Grad Student Descent to Automated Machine Learning (2019, background): “Certain opportunities for hybrid models in which humans and machines interact (for explaining failures [90] or intervening operations [91]) towards better AI accountability are also proposed in recent studies.”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, methodology): “User Understanding of Model Interview ([40]) and Self-explanation ([20, 47, 164]) Likert-scale Questionnaire ([99, 111, 113, 124, 133, 171]) Model Output Prediction User Prediction of Model Output ([96, 172, 173]) Model Failure Prediction User Prediction of Model Failure ([15, 161])”
Software Engineering for Machine Learning: A Case Study (2019, background): “Understanding when and how models fail to make accurate predictions is an active research area [29], [30], [31], which is attracting more attention as ML algorithms and optimization techniques become more complex.”
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance (2019, unknown): “Recent work (Nushi, Kamar, and Horvitz 2018), uses decision trees to predict and visualize error boundaries for the purpose of debugging ML models but more work is needed on deploying and evaluating such tools for decision-making.”
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use (2019, background): “An additional challenge is the fact that even models performing acceptably on average can have significant individual level errors (Nushi et al., 2018) which is undesirable in clinical practice.”