HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling
By Xiaosong Zhang, Yunjie Tian, Wei Huang, Qixiang Ye, Qi Dai, Lingxi Xie, Qi Tian. 2022
46 citations recorded by the source corpus.
Recently, masked image modeling (MIM) has offered a new methodology of self-supervised pre-training of vision transformers. A key idea of efficient implementation is to discard the masked image patches (or tokens) throughout the target network (encoder), which requires the encoder to be a plain vision transformer (e.g., ViT), albeit hierarchical vision transformers (e.g., Swin Transformer) have potentially better properties in formulating vision inputs. In this paper, we offer a new design of hierarchical vision transformers named HiViT (short for Hierarchical ViT) that enjoys both high efficiency and good performance in MIM. The key is to remove the unnecessary"local inter-unit operations", deriving structurally simple hierarchical vision transformers in which mask-units can be serialized like plain vision transformers. For this purpose, we start with Swin Transformer and (i) set the masking unit size to be the token size in the main stage of Swin Transformer, (ii) switch off inter-unit self-attentions before the main stage, and (iii) eliminate all operations after the main stage. Empirical studies demonstrate the advantageous performance of HiViT in terms of fully-supervised, self-supervised, and transfer learning. In particular, in running MAE on ImageNet-1K, HiViT-B reports a +0.6% accuracy gain over ViT-B and a 1.9$\times$ speed-up over Swin-B, and the performance gain generalizes to downstream tasks of detection and segmentation. Code will be made publicly available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Zhenlin Xu
- Marc Niethammer
- John Seon Keun Yi
- Qin Liu
- Kang Zhang
- In-So Kweon
- Chaoning Zhang
- Gedas Bertasius
In Computer Vision and Pattern Recognition
- In-So Kweon
- Gedas Bertasius
- Luc Van Gool
- Qin Liu
- Marc Niethammer
- Zhenlin Xu
- John Seon Keun Yi
- Kang Zhang
In Neurology
In Model Architectures
Citations in the corpus
- SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, methodology): “Hierarchical ViT with MAE Recently, HiViT [50] shows that hierarchical ViT can also be pretrained as MAE.”
- Context Autoencoder for Self-supervised Representation Learning (2022, unknown): “…that, MIM methods have developed rapidly, e.g., extended to frequency/semantic domain [87,61,84,58], combined with contrastive self-superivsed learning [72,49,94,47], efficient pretraining [101,46,13], mask strategy design [50, 54,57], scalability of MIM [92], and interpretation of MIM [89,56,52].”
- A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “Several works [51], [52], [53] attempt to boost the power of hViTs while achieving efficiency in MAE.”