Machine Learning Testing: Survey, Landscapes and Horizons

By Jie M. Zhang, Mark Harman, Lei Ma, Yang Liu. IEEE Transactions on Software Engineering, 2019

884 citations recorded by the source corpus.

This paper provides a comprehensive survey of techniques for testing machine learning systems; Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/218062f45c15f39bc8f4fb2c930ddf20b5809b11

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Bridging the Gap Between Ethics and Practice (2020, background): “[126] describe five typical tasks for machine learning, which may need distinctive workflows: “(1) Classification: to assign a category to each data instance, e.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “Behavior-based alternatives to the benchmarking paradigm, such as test suites [1, 126, 170], for example, can re-orient ML evaluation away from its current focus on the competitive determination of “state of the art”, and more towards an exploratory and descriptive probing of model capabilities…”
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021, background): “There is extensive literature on ML testing for detecting differences between the actual and expected behaviour of ML pipelines; for a survey, see [129].”
Trustworthy AI: From Principles to Practices (2021, background): “Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, methodology): “To cover this, we evaluate them for correctness, robustness, and fairness [48].”
Leveraging Automated Unit Tests for Unsupervised Code Translation (2021, methodology): “More generally, software testing tools have been largely ignored by the machine learning community (Zhang et al., 2020).”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “2) Challenges in Test Oracle Problem: The oracle problem has been a longstanding challenge in various testing applications, e.g., testing machine learning systems [148] and testing deep learning libraries [59].”
Software Engineering for AI-Based Systems: A Survey (2021, background): “[222] 138 studies (2007-2019) correctness, model relevance, robustness, security, data privacy, efficiency, fairness, interpretability testing workflow, testing components, testing properties, application scenarios 4 testing challenges categories: test input generation, test assessment criteria, ora”