Is Deep Learning Safe for Robot Vision? Adversarial Examples Against the iCub Humanoid

By Marco Melis, Ambra Demontis, Battista Biggio, Gavin Brown, Giorgio Fumera, Fabio Roli. 2017

102 citations recorded by the source corpus.

Deep neural networks have been widely adopted in recent years, exhibiting impressive performances in several application domains. It has however been shown that they can be fooled by adversarial examples, i.e., images altered by a barely-perceivable adversarial noise, carefully crafted to mislead classification. In this work, we aim to evaluate the extent to which robot-vision systems embodying deep-learning algorithms are vulnerable to adversarial examples, and propose a computationally efficient countermeasure to mitigate this threat, based on rejecting classification of anomalous inputs. We then provide a clearer understanding of the safety properties of deep networks through an intuitive empirical analysis, showing that the mapping learned by such networks essentially violates the smoothness assumption of learning algorithms. We finally discuss the main limitations of this work, including the creation of real-world adversarial examples, and sketch promising research directions.

Subfields: Artificial Intelligence, Molecular Biology, Safety, Robustness & Interpretability, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Safety, Robustness & Interpretability

Citations in the corpus

Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, methodology): “[63] demonstrated the vulnerability of robots to the adversarial manipulations of the input images using the techniques in [22].”
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples (2018, background): “Besides, recent studies have also conducted adversarial attacks in the real-world scenarios, including attacking road signs [28, 29, 24, 25], cell-phone cameras [30], robots [31], and generating 3D-printed adversarial objects [32].”
Towards Adversarial Malware Detection (2018, background): “The aforementioned optimization problem is usually solved by projected gradient descent [8, 9], and it has been used in follow-up work to generate adversarial PDF malware [83, 96], adversarial binaries [51] and adversarial images against deep networks [66].”
Review of Artificial Intelligence Adversarial Attack and Defense Technologies (2019, methodology): “[36] used the technology in [6] to prove the vulnerability of the robots to the input image under the adversarial operation.”
secml: A Python Library for Secure and Explainable Machine Learning (2019, methodology): “We also consider a variant of our PGD attack, referred to as PGD-patch here, where we restrict the attacker to only change the pixels of the image corresponding to the license plate, using a box constraint (Melis et al., 2017).”
Feature-Guided Black-Box Safety Testing of Deep Neural Networks (2017, methodology): “[25] replaces the softmax layer in a deep network with a multiclass SVM and then finds adversarial examples by performing a gradient computation.”