Human model evaluation in interactive supervised learning

By Rebecca Fiebrink, Perry R. Cook, D. Trueman. 2011

240 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Evaluation & Benchmarks, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Evaluation & Benchmarks

In Human-Computer Interaction

Citations in the corpus

Refining Language Models with Compositional Explanations (2021, background): “active learning [39]) or to determine which model to use [9].”
Toward User-Driven Sound Recognizer Personalization with People Who Are d/Deaf or Hard of Hearing (2021, background): “, [20]) could provide insight into how an individual’s samples shape the model, while user-defned decision boundaries (e.”
Power to the People: The Role of Humans in Interactive Machine Learning (2014, background): “In a follow-up investigation with a professional cellist (Fiebrink, Cook, and Trueman 2011), the cellist identified flaws in her playing technique while trying to train a gesture recognizer.”
Software Engineering for AI-Based Systems: A Survey (2021, background): “com [21], AI model criteria relevant for end users [56], automatic version control in notebooks [98], or practices collected via practitioner surveys and/or interviews [203, 224].”
Principles of Explanatory Debugging to Personalize Interactive Machine Learning (2015, background): “…have explored using this cycle of quick interactions to train instance-based classiﬁers [5, 13], enable better model selection by end users [2, 12, 40], elicit labels for the most important instances (e.g., active learning) [7, 37], and to improve reinforcement learning for automated agents…”
Human-in-the-loop machine learning: a state of the art (2022, methodology): “Fiebrink et al. (2011) already stated that in an IML system the evaluation of the models should go beyond their accuracy and should include subjective judgments of properties such as cost, confidence, complexity, etc. AL focuses on building better models in an algorithm-centered evaluation, but in…”
ModelTracker: Redesigning Performance Analysis Tools for Machine Learning (2015, background): “A similar strategy is direct evaluation , where users provide new data to the model to examine its predictions in real time (e.g., [9,10,21]).”
A Review of User Interface Design for Interactive Machine Learning (2018, background): “Similar approaches have demonstrated the potential for IML to support unique forms of music composition (Fiebrink et al. 2011; Tsandilas et al. 2009) and interactive experiences (Kleinsmith and Gillies 2013; Brenton et al. 2014; Gillies et al. 2015).”
Tell me more?: the effects of mental model soundness on personalizing an intelligent agent (2012, result): “This is in contrast to recent work in interactive machine learning, which has found that for some systems (e.g., gesture recognition frameworks), repeated use taught people the most salient aspects of how the system worked [7].”