On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems

By Besmira Nushi, Ece Kamar, Eric Horvitz, Donald Kossmann. 2016

83 citations recorded by the source corpus.

We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components. Understanding and fixing errors that arise in such integrative systems is difficult as failures can occur at multiple points in the execution workflow. Moreover, errors can propagate, become amplified or be suppressed, making blame assignment difficult. We propose a human-in-the-loop methodology which leverages human intellect for troubleshooting system failures. The approach simulates potential component fixes through human computation tasks and measures the expected improvements in the holistic behavior of the system. The method provides guidance to designers about how they can best improve the system. We demonstrate the effectiveness of the approach on an automated image captioning system that has been pressed into real-world use.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, ML Systems & Hardware, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In ML Systems & Hardware

In Human-Computer Interaction

Citations in the corpus

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, neutral): “components irrespective of previous failure [50].”
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, background): “…importance of explicitly considering biases that may come from the humans embedded at various stages of the ML development and maintenance pipeline, such as crowdworkers who annotate training data or user-study participants tasked with surfacing undesirable biases in ML systems [9, 47, 56, 78, 92].”
Software Engineering for Machine Learning: A Case Study (2019, background): “For example, previous work in this field [18], [19] found that variation in the inherent uncertainty (and error) of data-driven learning algorithms and complex component entanglement caused by hidden feedback loops could impose substantial changes (even in specific stages) which were previously well”
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers (2018, unknown): “t image-based techniques stemming from the AI communities [4], [10], [68], [69], [70]. 4.2 Debugging & Improving Models Building machine learning models is an iterative design process [71], [72], [73], and developing deep neural net7 works is no different. While mathematical foundations have bee”
An Empirical Analysis of Backward Compatibility in Machine Learning Systems (2020, methodology): “ncesfromaretrained model are used by other components of a larger system. The challenge of understanding how changes in one component propagate through an entire system has attracted recent interest. [2, 32, 36] discuss how hidden data and feature dependencies can cause component entanglement, unexp”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “1), test input or test oracle design [116], [120], [121], [123], [130], [154], [162], and test tool design [165], [167], [182] (see more in Section 5).”
A survey of deep learning techniques for autonomous driving (2019, methodology): “[142] describes a technique for debugging misclassifications due to bad training data, while an approach for troubleshooting faults due to complex interactions between linked machine learning components is proposed in [143].”
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure (2018, methodology): “For these purposes, we leverage the task design and dataset we collected in prior work for troubleshooting the image captioning sys-tem (Nushi et al. 2017).”