TAP-Vid: A Benchmark for Tracking Any Point in a Video

By Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, Yezhou Yang. Neural Information Processing Systems, 2022

317 citations recorded by the source corpus.

Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences about 3D shape, physical properties and object interactions. While the problem of tracking arbitrary physical points on surfaces over longer video clips has received some attention, no dataset or benchmark for evaluation existed, until now. In this paper, we first formalize the problem, naming it tracking any point (TAP). We introduce a companion benchmark, TAP-Vid, which is composed of both real-world videos with accurate human annotations of point tracks, and synthetic videos with perfect ground-truth point tracks. Central to the construction of our benchmark is a novel semi-automatic crowdsourced pipeline which uses optical flow estimates to compensate for easier, short-term motion like camera shake, allowing annotators to focus on harder sections of video. We validate our pipeline on synthetic data and propose a simple end-to-end point tracking model TAP-Net, showing that it outperforms all prior methods on our benchmark when trained on synthetic data.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d81c7c08765add0fd478aa3d6b3a9f4b7c1003a8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (2023, background): “Most of these prior approaches [6, 7, 12, 16, 47] are deep learning based and work by being trained on a large-dataset of ground-truth point tracks (often using synthetic training data).”
CoTracker: It is Better to Track Together (2023, methodology): “9 AJ (a) to (f)); (2) when tracking single target points, adding support points in a global grid significantly boosts performance ( +15 AJ (b) to (c)); (3) the latter is further improved by using a global/local grid ( +5 .”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, methodology): “Table 7: Static dummy baseline vs TapNet [19] results (average Jaccard, higher is better) for the point tracking task on the validation set.”
CoTracker3: Simpler and Better Point Tracking by Pseudo-Labeling Real Videos (2025, unknown): “We conduct our evaluation on TAP-Vid [6], comprising TAP-Vid-Kinetics, TAP-Vid-DAVIS, and RGB-Stacking.”
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement (2023, methodology): “1, we find that TAPIR improves over prior works by a large margin, as measured by performance on the TAP-Vid benchmark [13].”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, methodology): “MPI Sintel [13] Flyingthings++ [40, 25] Kubric [24] TAP-Vid-Kinetics [16] TAP-Vid-DAVIS [16] Depth & normals Table 1: Comparison of point tracking datasets.”