CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
By Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, Lin Tan. 2019
214 citations recorded by the source corpus.
Deep learning (DL) systems are widely used in domains including aircraft collision avoidance systems, Alzheimer's disease diagnosis, and autonomous driving cars. Despite the requirement for high reliability, DL systems are difficult to test. Existing DL testing work focuses on testing the DL models, not the implementations (e.g., DL software libraries) of the models. One key challenge of testing DL libraries is the difficulty of knowing the expected output of DL libraries given an input instance. Fortunately, there are multiple implementations of the same DL algorithms in different DL libraries. Thus, we propose CRADLE, a new approach that focuses on finding and localizing bugs in DL software libraries. CRADLE (1) performs cross-implementation inconsistency checking to detect bugs in DL libraries, and (2) leverages anomaly propagation tracking and analysis to localize faulty functions in DL libraries that cause the bugs. We evaluate CRADLE on three libraries (TensorFlow, CNTK, and Theano), 11 datasets (including ImageNet, MNIST, and KGS Go game), and 30 pre-trained models. CRADLE detects 12 bugs and 104 unique inconsistencies, and highlights functions relevant to the causes of inconsistencies for all 104 unique inconsistencies.
Subfields: Artificial Intelligence, Software, ML Systems & Hardware, Software Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Soham De
- Andrew Brock
- Samuel L. Smith
- Quoc V. Le
- Karen Simonyan
- Mohit Bansal
- Yoshua Bengio
- Lingming Zhang
In Software
In ML Systems & Hardware
In Software Engineering
Citations in the corpus
- High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “A number of these errors are identified by Pham et al. (2019), an academic paper on automated testing which discovers two such implementation errors in Keras and one in the CNTK toolkit.”
- Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2022, background): “CRADLE [55] is one of the first work in the area that detects inconsistencies by running existing models on multiple low-level backends of Keras [36].”
- Characterizing signal propagation to close the performance gap in unnormalized ResNets (2021, background): “BatchNorm also adds memory overhead (Rota Bul`o et al., 2018), and is a common source of implementation errors (Pham et al., 2019).”