An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

By Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, Baobao Chang. European Conference on Computer Vision, 2024

629 citations recorded by the source corpus.

In this study, we identify the inefficient attention phenomena in Large Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5, QwenVL-Chat and Video-LLaVA. We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs, suggesting a need for a sparser approach compared to textual data handling. To this end, we introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency by learning adaptive attention patterns in early layers and pruning visual tokens in subsequent ones. Our evaluations demonstrate FastV's ability to dramatically reduce computational costs (e.g., a 45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a wide range of image and video understanding tasks. The computational efficiency and performance trade-off of FastV are highly customizable and pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve a lower budget than that of a 7B-parameter model, while still maintaining superior performance. We believe FastV has practical values for deployment of LVLMs in edge devices and commercial models. Code is released at https://github.com/pkunlp-icler/FastV.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/3421d12d98956c36c5d39c0ef023b378b6922068

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Kurt Keutzer
2. Bo Li
3. Yuanhan Zhang
4. Jingkang Yang
5. Shuzheng Si
6. Haozhe Zhao
7. Zefan Cai
8. Liang Chen

In Radiology, Nuclear Medicine and Imaging

1. Kurt Keutzer
2. Jingkang Yang

Citations in the corpus

• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling (2024, background): “Interpretation of LLMs Prior research has shown that attention matrices in LLMs are typically sparse [3, 32, 34], focusing disproportionately on a few tokens.”
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, positive): “From left to right, we visualize the results after we apply token pruning to different layers.”
• Long Context Transfer from Language to Vision (2024, methodology): “Alternative approaches [11, 62, 31, 91] employ heuristic techniques to prune or merge the visual features.”
• MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, background): “Recent VLMs (Zhu et al., 2023; Liu et al., 2023b; Li et al., 2022; Alayrac et al., 2022; Dai et al., 2023; Chen et al., 2024b) have been proven effective for aligning visual inputs and frozen LLMs to obtain cross-modal generalization ability.”