Beat the Machine: Challenging Humans to Find a Predictive Model's “Unknown Unknowns”

By Josh Attenberg, Panagiotis G. Ipeirotis, Foster J. Provost. 2015

115 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Francisco Herrera
2. Salvador Garcia
3. Daniel Molina
4. Javier Del Ser
5. Homa Alemzadeh
6. Adrien Bennetot
7. Alejandro Barredo Arrieta
8. Sergio Gil-Lopez

In Information Systems

1. Salvador Garcia
2. Francisco Herrera
3. Kush R. Varshney
4. Homa Alemzadeh

In Evaluation & Benchmarks

1. Salvador Garcia
2. Daniel Molina
3. Francisco Herrera
4. Kush R. Varshney
5. Natalia Díaz Rodríguez
6. Homa Alemzadeh

In Safety, Robustness & Interpretability

1. Adrien Bennetot
2. Francisco Herrera
3. Kush R. Varshney
4. Natalia Díaz Rodríguez
5. Raja Chatila
6. Homa Alemzadeh
7. Salvador Garcia
8. Javier Del Ser

Citations in the corpus

• CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, neutral): “Examples for this include 368 [39, 40, 41, 12, 42, 34, 43, 38, 44], with recent works featuring re-training during dataset collection 369 (Beat-the-AI [22], StrategyQA [23] and Dynabench [21]).”
• Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “In this case, the inspection of the share of epistemic uncertainty (namely, the uncertainty due to lack of knowledge) of the input data and its correspondence with the model’s output confidence can inform the user and eventually trigger his/her rejection of the model’s output [370,371] .”
• On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products (2016, background): “In these pa rts of the space, distance from the decision boundary is fairly meaningless and the typical trigger for the reject option sh ould be avoided [40].”
• Engineering safety in machine learning (2016, background): “In these pa rts of the space, distance from the decision boundary is fairly meaningless and the typical trigger for the reject option sh ould be avoided [19].”
• Data Science of the People , for the People , by the People : A Viewpoint on an Emerging Dichotomy (2015, methodology): “When predictive models trained under expected error make gross mistakes, they are usually on people with rare characteristics; there are learning approaches to deal with rare classes [48], but there is little work on data points with rare features [58, 5].”