Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference

By Ting-Kuei Hu, Tianlong Chen, Haotao Wang, Zhangyang Wang. 2020

89 citations recorded by the source corpus.

Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) (Tsipras et al., 2019). Such a dilemma is shown to be rooted in the inherently higher sample complexity (Schmidt et al., 2018) and/or model capacity (Nakkiran, 2019), for learning a high-accuracy and robust classifier. In view of that, give a classification task, growing the model capacity appears to help draw a win-win between accuracy and robustness, yet at the expense of model size and latency, therefore posing challenges for resource-constrained applications. Is it possible to co-design model accuracy, robustness and efficiency to achieve their triple wins? This paper studies multi-exit networks associated with input-adaptive efficient inference, showing their strong promise in achieving a “sweet point" in co-optimizing model accuracy, robustness, and efficiency. Our proposed solution, dubbed Robust Dynamic Inference Networks (RDI-Nets), allows for each input (either clean or adversarial) to adaptively choose one of the multiple output layers (early branches or the final one) to output its prediction. That multi-loss adaptivity adds new variations and flexibility to adversarial attacks and defenses, on which we present a systematical investigation. We show experimentally that by equipping existing backbones with such robust adaptive inference, the resulting RDI-Nets can achieve better accuracy and robustness, yet with over 30% computational savings, compared to the defended original models.

Subfields: Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

1. Canwen Xu
2. Julian McAuley
3. Furu Wei
4. Tao Ge
5. Wangchunshu Zhou
6. Shiji Song
7. Aude Oliva
8. Gao Huang

In Optimization & Training

1. Julian McAuley
2. Canwen Xu
3. Shiji Song
4. Yulin Wang
5. Le Yang
6. Chongjian Ge
7. Sijia Liu
8. Aude Oliva

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, methodology): “For example, multi-exit models can be used to: 1) alleviate the over-thinking issue while reducing the overall computation [50], [266]; 2) perform long-tailed classification [267] by inducing early exiting in the training stage; and 3) improve the model robustness [268].”
• Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “Many adaptive computation methods are explored (Bengio et al., 2015; Wang et al., 2018; Graves, 2016; Hu et al., 2020; Wang et al., 2020b; Han et al., 2021b) to alleviate the computation burden.”
• ELF: An Early-Exiting Framework for Long-Tailed Classification (2020, background): “On the other hand [29, 30, 31] defines hardness based on prediction confidence.”
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning (2020, background): “[10, 16] showed AT is also effective in compressing or accelerating models [42] while preserving learned robustness.”
• BERT Loses Patience: Fast and Robust Inference with Early Exit (2020, background): “Recent studies [5, 27] attribute the vulnerability partially to the overthinking problem, arguing that it can be mitigated by early exit mechanisms.”
• AdderNet: Do We Really Need Multiplications in Deep Learning? (2019, unknown): “[14] proposed dubbed Robust Dynamic Inference Networks (RDI-Nets), which allows for each input to adaptively choose one of the multiple output layers to output its prediction.”
• IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “To overcame this issue, many adaptive computation methods are explored during the inference stage [1, 2, 57, 15, 26, 58, 20].”
• A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inference (2020, methodology): “To this end, we find that examples crafted by prior evasion attacks (Madry et al., 2017; Hu et al., 2020) fail to bypass the victim model’s early exits, and we show that an adversary can adapt such attacks to the goal of model slowdown by modifying its objective function.”