You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems

By Alex Groce, Todd Kulesza, Chaoqiang Zhang, Shalini Shamasunder, Margaret M. Burnett, Weng-Keen Wong, Simone Stumpf, Shubhomoy Das, Amber Shinsel, Forrest Bice and 1 more. 2014

76 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Software

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Software

Citations in the corpus

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, background): “cover a range of topics (Ballas, 2007; Bisantz, Finger, Seong, and Llinas, 1999; Blokpoel et al., 2012; Cahour and Forzy, 2009; Clarke and Smyth, 1993; Davies, 2011; Dunbar, 1995; Frank et al., 2016; Groce et al., 2014; Hoffman, 2013, 2017b; Johnson and Johnson, 1993; Kilgore and Voshell, 2014, 2014”
Principles of Explanatory Debugging to Personalize Interactive Machine Learning (2015, background): “We say “debugging” because personalizing a learning system is an end-user debugging problem: as discussed in [14], the user is trying to identify and correct faults in the system’s reasoning that are responsible for predictions that failed to meet the user’s expectations.”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “with the agent’s output, as well as on users’ conﬁdence and experience with the system. Another use case for machine learning explanations is to help users to justify the correctness of system output [39,61,116]. Explanations also assist users in debugging interactive machine learning programs for t”
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure (2018, methodology): “In the context of system performance analysis, human input has been leveraged for system evaluation (Steinfeld et al. 2007; Shinsel et al. 2011) and testing (Groce et al. 2014; Attenberg, Ipeirotis, and Provost 2011).”
Six Human-Centered Artificial Intelligence Grand Challenges (2023, background): “This may result in an inappropriate shift in responsibility where end users are expected to evaluate a tool ’ s performance without a desire or realistic capability of doing so (Amershi et al., 2014; Groce et al., 2014).”
A Review of User Interface Design for Interactive Machine Learning (2018, background): “In the case of a user training a model to capture their own personal concept, Groce et al. (2014) highlight that potentially only that particular individual can properly evaluate the quality of their model.”
Association for Computational Linguistics " Why Should I Trust You? " Explaining the Predictions of Any Classifier (2016, unknown): “ a decision should not be trusted. Furthermore, they assume that the current evaluation metrics are reliable, which may not be the case if problems such as data leakage are present. Other recent work [10] focuses on exposing users to different kinds of mistakes (our pick step). Interestingly, the su”