Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding

By Zizhao Zhang, Long Zhao, Ting-Li Chen, Sercan Ö. Arik, Tomas Pfister. 2021

227 citations recorded by the source corpus.

Hierarchical structures are popular in recent vision transformers, however, they require sophisticated designs and massive datasets to work well. In this paper, we explore the idea of nesting basic local transformers on non-overlapping image blocks and aggregating them in a hierarchical way. We find that the block aggregation function plays a critical role in enabling cross-block non-local information communication. This observation leads us to design a simplified architecture that requires minor code changes upon the original vision transformer. The benefits of the proposed judiciously-selected design are threefold: (1) NesT converges faster and requires much less training data to achieve good generalization on both ImageNet and small datasets like CIFAR; (2) when extending our key ideas to image generation, NesT leads to a strong decoder that is 8 times faster than previous transformer-based generators; and (3) we show that decoupling the feature learning and abstraction processes via this nested hierarchy in our design enables constructing a novel method (named GradCAT) for visually interpreting the learned model. Source code is available https://github.com/google-research/nested-transformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Prompt for Continual Learning (2021, methodology): “Our method is based on pre-trained ViT-B/16 [11, 67], which has become a common asset in advanced vision communities.”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “Inspired by the competitive performance and global receptive ﬁeld of transformer models, follow-up works are proposed to reﬁne the architecture [36, 37], explore the relationship between CONV nets and ViT [38, 39, 40], and adapt ViT to different computer vision tasks [13, 41, 42, 43, 44, 45, 46].”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, background): “With appropriate changes, ViTs demonstrate strong performance in downstream vision tasks [83, 90, 89, 42, 41, 22, 87].”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, background): “More sophisticated encoders are possible, to leverage recent advances in architecture designs [20, 41, 25, 71, 26], but this is not our main focus so we opt for simplicity.”
Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning (2022, result): “Similar to our work is NesT and Hierarchical Perciever, which similarly partitions and then aggregates features from non-overlapping image regions via Transformer blocks [14, 83].”