MixFormer: End-to-End Tracking with Iterative Mixed Attention

By Yutao Cui, Jiang Cheng, Limin Wang, Gangshan Wu. Computer Vision and Pattern Recognition, 2022

836 citations recorded by the source corpus.

Tracking often uses a multistage pipeline of feature extraction, target information integration, and bounding box estimation. To simplify this pipeline and unify the process of feature extraction and target information integration, we present a compact tracking framework, termed as MixFormer, built upon transformers. Our core design is to utilize the flexibility of attention operations, and propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. This synchronous modeling scheme allows to extract target-specific discriminative features and perform extensive communication between target and search area. Based on MAM, we build our MixFormer tracking framework simply by stacking multiple MAMs with progressive patch embedding and placing a localization head on top. In addition, to handle multiple target templates during online tracking, we devise an asymmetric attention scheme in MAM to reduce computational cost, and propose an effective score prediction module to select high-quality templates. Our MixFormer sets a new state-of-the-art performance on five tracking benchmarks, including LaSOT, TrackingNet, VOT2020, GOT-10k, and UAV123. In particular, our MixFormer-L achieves NP score of 79.9% on LaSOT, 88.9% on TrackingNet and EAO of 0.555 on VOT2020. We also perform in-depth ablation studies to demonstrate the effectiveness of simultaneous feature extraction and information integration. Code and trained models are publicly available at https://github.com/MCG-NJU/MixFormer.

Subfields: Computer Vision and Pattern Recognition, Global and Planetary Change, Safety, Risk, Reliability and Quality

View Paper: https://www.semanticscholar.org/paper/b6eaec7917439d79ce840fa97bc371552e9b6685

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Dacheng Tao
4. Xiaogang Wang
5. Andrea Vedaldi
6. Huchuan Lu
7. Yi Yang
8. Andrew Zisserman

In Global and Planetary Change

1. Huchuan Lu
2. Houwen Peng
3. Bineng Zhong
4. Dong Wang

In Safety, Risk, Reliability and Quality

1. Houwen Peng
2. Huchuan Lu
3. Dong Wang
4. Xin Chen
5. Bineng Zhong

Citations in the corpus

• HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, background): “Recently, several efforts have been made to integrate the two types of blocks into a unified model with a sequential [12, 28, 64, 59] or parallel [43, 11] design.”
• SeqTrack: Sequence to Sequence Learning for Visual Object Tracking (2023, methodology): “Prior knowledge, such as window penalty [2,27] and online update [12, 28, 53], has been widely incorporated into existing tracking models and proved effective [7,10,43,55].”
• CoTracker: It is Better to Track Together (2023, background): “We show that, by training the network by unrolling several sliding windows, CoTracker achieves excellent long-term tracking performance as well.”
• Autoregressive Visual Tracking (2023, methodology): “Template Z could also be renewed at each time step with an updating mechanism [13, 56], or simply being the initial one [40, 64].”
• ODTrack: Online Dense Temporal Token Learning for Visual Tracking (2024, methodology): “Video Sequence Sampling Strategy Most existing trackers (Yan et al. 2021a; Cui et al. 2022; Ye et al. 2022) commonly sample image-pairs within a short-term interval, such as 50, 100, or 200 frame intervals.”