Token Merging: Your ViT But Faster

By Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, Judy Hoffman. International Conference on Learning Representations, 2022

1,128 citations recorded by the source corpus.

We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm that is as fast as pruning while being more accurate. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case. ToMe can also easily be applied during training, improving in practice training speed up to 2x for MAE fine-tuning on video. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop. Qualitatively, we find that ToMe merges object parts into one token, even over multiple frames of video. Overall, ToMe's accuracy and speed are competitive with state-of-the-art on images, video, and audio.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1dff6b1b35e2d45d4db57c8b4e4395486c3e365f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Open-Vocabulary Object Detection (2023, background): “Various forms of token dropping or pooling have been proposed to improve efficiency [19, 26, 35, 20, 2].”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, positive): “And then we use the dot product between keys to calculate which tokens have similar visual information [4]: which yields KK T ( i, j ) for tokens i, j in vectorized form for the set of all tokens 1 , 2 , · · · , n , where n is the number of input visual tokens.”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “There have also been recent methods [6, 39, 49, 63, 68] to explore the role of dynamic token within the transformer framework.”
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, methodology): “…compression methods can be grouped into three main categories: (1) Token pruning (Goyal et al., 2020; Kim and Cho, 2021; Modar-ressi et al., 2022) and token merging (Bolya et al., 2023), which need model fine-tuning or intermediate results during inference and have been used with BERT-scale models.”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “Recent methods has explored dynamic image tokens (Ma et al., 2023; Xu et al., 2022; Bolya et al., 2022) or video tokens (Lee et al., 2024; Ren et al., 2023a; Choi et al., 2024) within the Transformer (Vaswani, 2017) framework.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “For the second concern, an intuitive approach is to drop the similar tokens [42, 43, 33, 44] or, like PNP, apply dropping rate schedulers.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, methodology): “This method transforms the dense to-kens to the sparse memories, which are stored in long-term memory.”
FastComposer: Tuning-Free Multi-subject Image Generation with Localized Attention (2023, neutral): “Naively applying subject-augmented conditioning leads to subject overﬁtting (Figure 2 right), re-stricting the user’s ability to edit subjects based on textual directives.”
SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, neutral): “One focuses on the matching algorithm using the Bipartite Soft Matching (BSM) (Bolya et al., 2022).”