IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers

By Bowen Pan, Rameswar Panda, Zhangyang Wang, Rogério Schmidt Feris, Aude Oliva. 2021

204 citations recorded by the source corpus.

The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. In spite of the impressive success made by transformers in a variety of vision tasks, it still suffers from heavy computation and intensive memory costs. To address this limitation, this paper presents an Interpretability-Aware REDundancy REDuction framework (IA-RED$^2$). We start by observing a large amount of redundant computation, mainly spent on uncorrelated input patches, and then introduce an interpretable module to dynamically and gracefully drop these redundant patches. This novel framework is then extended to a hierarchical structure, where uncorrelated tokens at different stages are gradually removed, resulting in a considerable shrinkage of computational cost. We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4x speed-up for state-of-the-art models like DeiT and TimeSformer, by only sacrificing less than 0.7% accuracy. More importantly, contrary to other acceleration approaches, our method is inherently interpretable with substantial visual evidence, making vision transformer closer to a more human-understandable architecture while being lighter. We demonstrate that the interpretability that naturally emerged in our framework can outperform the raw attention learned by the original visual transformer, as well as those generated by off-the-shelf interpretation methods, with both qualitative and quantitative results. Project Page: http://people.csail.mit.edu/bpan/ia-red/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Zehua Sun
5. Wuyang Chen
6. Zhangyang Wang
7. Xing Sun
8. Changsheng Xu

In Computer Vision and Pattern Recognition

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Zehua Sun
5. Zhangyang Wang
6. Hengduo Li
7. Zuxuan Wu
8. Ser-Nam Lim

Citations in the corpus

• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, background): “producing policies on what patch size [47] and which patches [30, 33] to use conditioned on input image.”
• Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, methodology): “For overall comparisons with the state-of-the-art methods (Rao et al. 2021; Tang et al. 2021; Chen et al. 2021; Pan et al. 2021), we conduct the token selection and slow-fast token updating from the fifth layer of DeiT and the third layer (excluding the convolution layers) of LeViT, respectively.”
• Auto-scaling Vision Transformers without Training (2022, background): “Although recent works attempt to enhance ViT’s data and resource efficiency (Touvron et al., 2020; Hassani et al., 2021; Pan et al., 2021; Chen et al., 2021d), the heavy computation cost (e.g., quadratic with respect to the number of tokens) is still overwhelming, compared with training CNNs.…”
• Human Action Recognition From Various Data Modalities: A Review (2020, background): “Researchers thus have made plenty of efforts to decrease computation complexity [153], [156], [161], [164], [168], [176] and memory cost [154], [161], [164], [165], [176].”