Analysis of Invariance and Robustness via Invertibility of ReLU-Networks

By Jens Behrmann, Sören Dittmer, Pascal Fernsel, Peter Maass. 2018

13 citations recorded by the source corpus.

Studying the invertibility of deep neural networks (DNNs) provides a principled approach to better understand the behavior of these powerful models. Despite being a promising diagnostic tool, a consistent theory on their invertibility is still lacking. We derive a theoretically motivated approach to explore the preimages of ReLU-layers and mechanisms affecting the stability of the inverse. Using the developed theory, we numerically show how this approach uncovers characteristic properties of the network.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Statistical and Nonlinear Physics

Citations in the corpus

Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness (2019, methodology): “The cross-entropy loss as well as architectural elements such as ReLU activation functions have been put forward as possible sources of excessive invariance (Jacobsen et al., 2019; Behrmann et al., 2018).”
Excessive Invariance Causes Adversarial Vulnerability (2018, background): “The role of invariance has been studied extensively in the context of deep neural networks (Bruna & Mallat, 2013; Mallat, 2016; Achille & Soatto, 2018; Behrmann et al., 2018; Cadena et al., 2018).”
Deep Neural Networks are Surprisingly Reversible: A Baseline for Zero-Shot Inversion (2021, background): “For example, one direction of works provides theoretical proof of the invertibility assuming the weights of DNNs are random-like [19, 29, 30], and expansion rates are larger than a certain constant [21, 31].”