A Safety Framework for Critical Systems Utilising Deep Neural Networks

By Xingyu Zhao, Alec Banks, James Sharp, Valentin Robu, David Flynn, Michael Fisher, Xiaowei Huang. 2020

52 citations recorded by the source corpus.

Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative -- it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Wenjie Ruan
2. Youcheng Sun
3. Min Wu
4. Xiaowei Huang
5. Daniel Kroening
6. James Sharp
7. Emese Thamo
8. Xinping Yi

Citations in the corpus

• A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability (2018, background): “Research on this has started in [Zhao et al., 2020], where a Bayesian inference approach is taken.”