Machine Learning that Matters

By Kiri L. Wagstaff. 2012

342 citations recorded by the source corpus.

Much of current machine learning (ML) research has lost its connection to problems of import to the larger world of science and society. From this perspective, there exist glaring limitations in the data sets we investigate, the metrics we employ for evaluation, and the degree to which results are communicated back to their originating domains. What changes are needed to how we conduct research to increase the impact that ML has? We present six Impact Challenges to explicitly focus the field's energy and attention, and we discuss existing obstacles that must be addressed. We aim to inspire ongoing discussion and focus on ML that matters.

Subfields: Artificial Intelligence, Information Systems and Management, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems and Management

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “[Wagstaff, 2012, p.3] Wagstaff here calls out the fact that the task is not just poorly formulated but also likely measured inappropriately, in a way that makes the results difﬁcult to interpret meaningfully.”
Deep Reinforcement Learning that Matters (2017, methodology): “However, while the question of reproducibility and good experimental practice has been examined in related ﬁelds (Wagstaff 2012; Boulesteix, Lauer, and Eugster 2013; Stodden, Leisch, and Peng 2014; Bouckaert and Frank 2004; Bouckaert 2004; Vaughan and Wawerla 2012), to the best of our knowledge…”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “It is universally agreed that interpretability is a key element of trust for AI models (Wagstaff, 2012; Rudin and Wagstaff, 2014; Lo Piano, 2020; Ashoori and Weisz, 2019; Thiebes et al., 2020; Spiegelhalter, 2020; Brundage et al., 2020).”
On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products (2016, background): “that can be judged harmful or not [6].”
The Values Encoded in Machine Learning Research (2021, background): “We find a clear emphasis on technical novelty, rather than critique of past work, or demonstration of measurable progress on societal problems, as has previously been observed [63].”
Increasing Trust in AI Services through Supplier's Declarations of Conformity (2018, background): “Abstract loss functions may be informed by real-world quality metrics [31], including context-dependent ones [32].”
Likelihood Ratios for Out-of-Distribution Detection (2019, background): “The sequential nature of genetic sequences provides a new modality and hopefully encourages the OOD research community to contribute to “machine learning that matters” (Wagstaff, 2012).”
Predictive Analytics for City Agencies: Lessons from Children's Services (2017, neutral): “20 ACS wants to eventually use the repeat reports model to not only measure which children have a high probability of an adverse outcome but also understand features that are important in the calculation of these probabilities.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Wagstaff (2012) discusses the strong lack of real-world applications in ML conferences and the subsequent impact on research directions this can have.”
Machine learning for science and society (2013, background): “…applied topics between us include criminol-ogy (Wang et al. 2013), crop yield prediction (Wagstaff et al. 2008), the energy grid (Rudin et al. 2010, 2012), healthcare (Letham et al. 2013b; McCormick et al. 2012), information retrieval (Letham et al. 2013a), interpretable models (Letham et al.…”
A Plea for Neutral Comparison Studies in Computational Sciences (2012, background): “Similar discussions can be found in other fields of application of machine learning and computational statistics [2].”
The Limits of Abstract Evaluation Metrics: The Case of Hate Speech Detection (2017, background): “As a result, their real-world impact–"dollars saved, lives preserved, time conserved" [6]—is often unknown and may diverge depending on the specifics of each domain [3].”
On the Machine Learning of Ethical Judgments from Natural Language (2022, background): “In the case of a machine like Delphi, however, who is accountable is inherently obscured (Wagstaff, 2012).”
Data Science of the People , for the People , by the People : A Viewpoint on an Emerging Dichotomy (2015, background): “As has also been recognized by others, we believe the primary reason for this emerging dichotomy is the recognition that to make impact in certain applications of data science, the accuracy of the results is not the only criterion that matters [53, 42].”