Transformer in Transformer

By Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, Yunhe Wang. 2021

2,223 citations recorded by the source corpus.

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16$\times$16) as"visual sentences"and present to further divide them into smaller patches (e.g., 4$\times$4) as"visual words". The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, e.g., we achieve an 81.5% top-1 accuracy on the ImageNet, which is about 1.7% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at https://github.com/huawei-noah/CV-Backbones, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/TNT.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, background): “Some of them [4, 19, 61, 54, 79] introduce local dependency into vision transformers by modifying the patch embedding block or the transformer block or both, while others [21, 37, 57] adopt a pyramid structure to reduce the overall computation while maintaining the models’ ability to capture low-lev”
Towards Robust Vision Transformer (2021, methodology): “To extract low-level features of patches, CeiT [50], LeViT [12] and TNT [14] use a convolutional stem instead of the original linear layer, T2T-ViT [51] leverages self-attention to model dependencies among neighboring pixels.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “Concurrent to our work are some that modify the ViT architecture [66, 15, 25] for better image classification.”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “In [22, 13, 36, 38] the authors propose hierarchical versions of ViT.”
Transformers in Vision: A Survey (2021, background): “Transformer in Transformer [88] computes attention at two levels: patch-level (as done is standard ViTs [11]) and local sub-patch-level (e.”
Going deeper with Image Transformers (2021, methodology): “We compare CaiT with DeiT [64], Vit-B [19], TNT [26], T2T [75] and to several state-of-the-art convnets: Regnet [51] improved by Touvron et al.”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “More recent methods such as T2T ViT [53], CPVT [54], TNT [55], CrossViT [56] and LocalViT [57] introduce tailored changes to ViT to further improve image classiﬁcation performance.”
CvT: Introducing Convolutions to Vision Transformers (2021, methodology): “With fewer paramerters, CvT-13 achieves a 81.6% ImageNet Top-1 accuracy, outperforming PVT-Small [34], T2T-ViT t -14 [41], TNT-S [14] by 1.7%, 0.8%, 0.2% respectively.”
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, background): “Since then, ViT has instantly inspired several attempts to improve its efficiency and effectiveness from different aspects [35, 45, 14, 38, 19].”
PVT v2: Improved baselines with Pyramid Vision Transformer (2021, methodology): “TNT [11] utilizes inner and outer Transformer blocks to generate pixel and patch embed-dings respectively.”
XCiT: Cross-Covariance Image Transformers (2021, background): “Han et al. [27] introduce a transformer module for intra-patch structure, exploiting pixel-level information and integrating with an inter-patch transformer to attain higher representation power. d’Ascoli et al. [18] consider the initialization of self-attention blocks as a convolutional operator,…”
Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, methodology): “Transformer is initially applied to NLP for sequence modeling, recent works apply transformer to CV tasks to process images (Dosovitskiy et al., 2021; Touvron et al., 2021; Liu et al., 2021b; Chen et al., 2021; Han et al., 2021).”
Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers (2021, methodology): “TNT [43] employs a transformer suitable for fine-grained image tasks, which further divides the original image patch and conducts self-attention mechanism calculations in smaller units.”
Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism (2023, methodology): “In the field of object detection, by using the encoder-decoder structure with the transformer, researchers have constructed a series of DETR-like object detection models, such as DETR [3] and DINO [43].”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “Recently, some works try to introduce the intrinsic IB of CNNs into vision transformers explicitly [26,58,25,43,18,86, 80,91,9,49].”
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation (2021, background): “…compact, strong, and simple, avoid using any hierarchical design (“SD”: spatial downsampling, “MF”: multi-scale features, “2 × ”: double channels). extend transformers to more vision tasks, such as object detection [18,24,28], semantic segmentation [18,24,28,42] and video action recognition [1,12].”
A Survey of Transformers (2021, methodology): “[92], HIBERT[166], Liu and Lapata [86], Hi-Transformer[145] TENER[154], TNT[48]”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, neutral): “The comparing methods include DeiT (Touvron et al., 2021a), CaiT (Touvron et al., 2021b), LV-ViT (Jiang et al., 2021), CoaT (Xu et al., 2021), Swin (Liu et al., 2021), Twins (Chu et al., 2021), Visformer (Chen et al., 2021), ConViT (d’Ascoli et al., 2021), TNT (Han et al., 2021a), Efﬁcient-Net (Tan ”