Testing and Validating Machine Learning Classifiers by Metamorphic Testing

By Xiaoyuan Xie, Joshua Wing Kei Ho, Christian Murphy, Gail E Kaiser, Baowen Xu, Tsong Yueh Chen. Journal of Systems and Software, 2011

402 citations recorded by the source corpus.

Machine Learning algorithms have provided core functionality to many application domains - such as bioinformatics, computational linguistics, etc. However, it is difficult to detect faults in such applications because often there is no “test oracle” to verify the correctness of the computed outputs. To help address the software quality, in this paper we present a technique for testing the implementations of machine learning classification algorithms which support such applications. Our approach is based on the technique “metamorphic testing”, which has been shown to be effective to alleviate the oracle problem. Also presented include a case study on a real-world machine learning application framework, and a discussion of how programmers implementing machine learning algorithms can avoid the common pitfalls discovered in our study. We also conduct mutation analysis and cross-validation, which reveal that our method has high effectiveness in killing mutants, and that observing expected cross-validation result alone is not sufficiently effective to detect faults in a supervised classification program. The effectiveness of metamorphic testing is further confirmed by the detection of real faults in a popular open-source classification program.

Subfields: Information Systems, Software, Signal Processing

View Paper: https://www.semanticscholar.org/paper/0c1313b743d82c9901ef32a2a27df463bc2e9ae0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Mark Harman
2. Gordon Fraser
3. Lingming Zhang
4. Jie M. Zhang
5. Yang Liu
6. Lei Ma
7. Sergio Segura
8. Antonio Ruiz-Cortés

In Software

1. Lingming Zhang
2. Gordon Fraser
3. Mark Harman
4. Jie M. Zhang
5. Yang Liu
6. Lei Ma
7. Sergio Segura
8. Ana Belén Sánchez

In Signal Processing

1. Gordon Fraser
2. Sarfraz Khurshid
3. Mark Harman
4. Lei Ma
5. Lingming Zhang
6. Yang Liu
7. Jie M. Zhang

Citations in the corpus

• DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems (2018, methodology): “Metamorphic testing has been applied for testing machine learning classifiers [30, 40, 41].”
• Metamorphic Testing (2018, methodology): “The effectiveness of MT in alleviating the oracle problem has been shown repeatedly in numerous studies, covering many different domains, including bioinformatics [16, 75, 76, 80]; web services [14, 84]; embedded systems [13, 44, 49]; components [8, 59]; compilers [50, 86]; databases [52]; machine l”
• Machine Learning Testing: Survey, Landscapes and Horizons (2019, methodology): “In 2011, they further evaluated their approach using mutated machine learning code [241].”
• A Survey on Metamorphic Testing (2016, methodology): “[31], [109] proposed using metamorphic testing for the detection of faults in supervised classifiers.”
• Trustworthy Artificial Intelligence: A Review (2022, methodology): “Xie et al. [217] have tested different classifiers using this technique.”