CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries

By Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, Lin Tan. 2019

214 citations recorded by the source corpus.

Deep learning (DL) systems are widely used in domains including aircraft collision avoidance systems, Alzheimer's disease diagnosis, and autonomous driving cars. Despite the requirement for high reliability, DL systems are difficult to test. Existing DL testing work focuses on testing the DL models, not the implementations (e.g., DL software libraries) of the models. One key challenge of testing DL libraries is the difficulty of knowing the expected output of DL libraries given an input instance. Fortunately, there are multiple implementations of the same DL algorithms in different DL libraries. Thus, we propose CRADLE, a new approach that focuses on finding and localizing bugs in DL software libraries. CRADLE (1) performs cross-implementation inconsistency checking to detect bugs in DL libraries, and (2) leverages anomaly propagation tracking and analysis to localize faulty functions in DL libraries that cause the bugs. We evaluate CRADLE on three libraries (TensorFlow, CNTK, and Theano), 11 datasets (including ImageNet, MNIST, and KGS Go game), and 30 pre-trained models. CRADLE detects 12 bugs and 104 unique inconsistencies, and highlights functions relevant to the causes of inconsistencies for all 104 unique inconsistencies.

Subfields: Artificial Intelligence, Software, ML Systems & Hardware, Software Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Soham De
  2. Andrew Brock
  3. Samuel L. Smith
  4. Quoc V. Le
  5. Karen Simonyan
  6. Mohit Bansal
  7. Yoshua Bengio
  8. Lingming Zhang

In Software

  1. Yinlin Deng
  2. Chun Xia
  3. Lingming Zhang
  4. Haoran Peng
  5. Chenyuan Yang
  6. Ling-Ming Zhang
  7. Soren Dunn

In ML Systems & Hardware

  1. Lingming Zhang
  2. Soham De

In Software Engineering

  1. Karen Simonyan

Citations in the corpus