Taxonomy of Real Faults in Deep Learning Systems
By Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, Paolo Tonella. 2019
363 citations recorded by the source corpus.
The growing application of deep neural networks in safety-critical domains makes the analysis of faults that occur in such systems of enormous importance. In this paper we introduce a large taxonomy of faults in deep learning (DL) systems. We have manually analysed 1059 artefacts gathered from GitHub commits and issues of projects that use the most popular DL frameworks (TensorFlow, Keras and PyTorch) and from related Stack Overflow posts. Structured interviews with 20 researchers and practitioners describing the problems they have encountered in their experience have enriched our taxonomy with a variety of additional faults that did not emerge from the other two sources. Our final taxonomy was validated with a survey involving an additional set of 21 developers, confirming that almost all fault categories (13/15) were experienced by at least 50% of the survey participants.
Subfields: Computer Networks and Communications, Information Systems, Software
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Peter Kairouz
- Paul Ralph
- H. Brendan McMahan
- Sebastian Baltes
- Daniel Ramage
- Elad Hazan
- Francesco Orabona
- Satyen Kale
In Information Systems
- Rajiv Mathews
- Sebastian Baltes
- Paul Ralph
- H. Brendan McMahan
- Daniel Ramage
- Peter Kairouz
- Franccoise Beaufays
- Ananda Theertha Suresh
In Software
Citations in the corpus
- Generative Models for Effective ML on Private, Decentralized Datasets (2019, methodology): “Independently, Humbatova et al. (2019) identifies ‘text segmentation’ and ‘pixel encoding’ as being very common sources of bugs.”
- Sampling in software engineering research: a critical review and guidelines (2020, background): “Other repository mining studies clearly explain sampling was constrained by API limitations, and how the researchers worked around those constraints (e.g. Humbatova et al. 2020).”