DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

2021

Citations in the corpus

• Global Filter Networks for Image Classification (2021, background): “The transformer architecture, originally designed for the natural language processing (NLP) tasks [46], has shown promising performance on various vision problems recently [10, 43, 29, 53, 4, 51, 38, 5].”
• PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers (2021, background): “Now they begin their journey in computer vision [7, 22, 35, 29, 25, 30].”
• A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, methodology): “Processing an image with vision transformers remains computationally expensive, primarily due to the quadratic number of interactions between tokens [36,40,53].”
• Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “Table 7: Comparisons on EViT and DynamicViT (Rao et al., 2021). indicates the model is initialized with a pre-trained DeiT-S.”
• EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “…methodologies, such as proposing new architectures or modules [53, 54, 55, 56, 57, 58], re-thinking self-attention and sparse-attention mechanisms [59, 60, 61, 62, 63, 64, 65], and utilizing search algorithms that are widely explored in CNNs to find smaller and faster ViTs [66, 28, 29, 67].”
• Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, methodology): “Previous token pruning works (Rao et al. 2021; Tang et al. 2021) present a gradual shrinking architecture, in which more tokens are recognized as placeholder tokens in deeper layers.”
• Scaling Open-Vocabulary Object Detection (2023, methodology): “The image and text encoders are initialized from contrastively trained image-text models (CLIP, unless noted otherwise); the detection heads are randomly initialized.”
• Embracing Single Stride 3D Object Detector with Sparse Transformer (2021, background): “Many advanced techniques can be borrowed to enhance the performance of SST. (1) Better efficiency : There are a lot of techniques can be adopted to improve our efficiency, for example, token selection [42, 57], attention simplication [56].”
• CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, background): “Recent works have been focused on improving the architectures of attention models, including sparse attention [29, 30, 31, 32, 33, 34, 35], enlarged receptive fields [36, 37], pyramidal designs [38, 39, 40], efficient alternatives [41, 42, 43], etc.”
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, methodology): “Table 4: Comparisons between DVT and three state-of-the-art methods for improving the computational efficiency of vision Transformers, i.e., Data distillation [38], DynamicViT [32] and Patch slimming [36].”
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, unknown): “3.3.”