MOTS: Multi-Object Tracking and Segmentation

By Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, Bastian Leibe. 2019

664 citations recorded by the source corpus.

This paper extends the popular task of multi-object tracking to multi-object tracking and segmentation (MOTS). Towards this goal, we create dense pixel-level annotations for two existing tracking datasets using a semi-automatic annotation procedure. Our new annotations comprise 65,213 pixel masks for 977 distinct objects (cars and pedestrians) in 10,870 video frames. For evaluation, we extend existing multi-object tracking metrics to this new task. Moreover, we propose a new baseline method which jointly addresses detection, tracking, and segmentation with a single convolutional network. We demonstrate the value of our datasets by achieving improvements in performance when training on MOTS annotations. We believe that our datasets, metrics and baseline will become a valuable resource towards developing multi-object tracking approaches that go beyond 2D bounding boxes. We make our annotations, code, and models available at https://www.vision.rwth-aachen.de/page/mots.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “tracking and segmentation (MOTS) [52] task extends MultiObject Tracking (MOT) [48] from a bounding box level to a pixel level.”
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning (2018, methodology): “Although it may be feasible to label a large number of images with simple annotations such as drivable areas and object bounding boxes [11, 19], it remains challenging to obtain more complicated annotations such as instance segmentation [3], not to mention multi-object detection and segmentation tra”
Video Instance Segmentation (2019, neutral): “Recently, a multi-object tracking and segmentation dataset [32] is proposed to evaluate multi-object tracking along with instance segmentation.”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “The task of video object segmentation requires segmenting foreground objects in a class-agnostic fashion [43, 21, 36, 38], often by leveraging ground truth masks available for the first frame during inference [6, 32, 10, 19, 37].”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, methodology): “In order to construct our dataset and to inform our model prior, we extract segmentation masks from YouTube-VOS [17, 18] and KITTI-MOTS [19–21].”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “And like image segmentation, most existing methods are specialist models comprising multiple stages with pipe-lined frameworks, e.g., track-detect [67, 57, 36, 6, 44], clip-match [3, 5], propose-reduce [37].”
Opening up Open World Tracking (2021, methodology): “Each pair left: The standard approach to multi-object tracking is to detect, track and possibly segment objects that correspond to specific, pre-defined semantic classes, such as cars and pedestrians [78].”
Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (2023, neutral): “Traditionally video tracking algorithms have been focused on tracking whole objects (as bounding boxes or segmentation masks) [19, 30, 38], or tracking dense scene points but only between two timesteps (e.g. optical-flow / scene flow) [4, 37].”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, methodology): “Semi-supervised learning has been widely applied to several computer vision tasks, including semantic segmentation [57,19,59,31,80,35,81,73,96], object detection [68,76,64], instance segmentation [35,60], panoptic segmentation [45], human pose estimation [58], person re-identification [93], multi-ob”
Transformer-Based Visual Segmentation: A Survey (2023, background): “There are also several video segmentation tasks, including video object segmentation (VOS) [56], [98], referring video segmentation [57], multi-Object tracking, and segmentation (MOTS) [99].”
Segment as Points for Efficient Online Multi-Object Tracking and Segmentation (2020, methodology): “We compare recent works on MOTS: TRCNN [35], MOTSNet [28], BePix [32], and MOTSFusion (online) [18].”
3D Multi-Object Tracking: A Baseline and New Evaluation Metrics (2019, methodology): “raph matching problem and solve it using the Hungarian algorithm [29,33,34]. Beyond using the Hungarian algorithm in a post-processing step, modern online methods design the deep association networks [35,36] that are able to construct the association using neural networks. Our MOT system also belong”
Video Instance Segmentation Tracking With a Modified VAE Architecture (2020, methodology): “For a fair comparison, we re-train MaskTrack R-CNN [66] and TrackR-CNN [59] on our hardware platform, using the same ResNet-101 backbone and following their training and evaluation protocol.”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “To summarize, our contributions are the following: (i) We propose a unified approach for tasks involving instance segmentation in videos [11,85,100].”
Joint Object Detection and Multi-Object Tracking with Graph Neural Networks (2021, background): “[26]–[29] proposed to extend an object detector by adding a re-identification (Re-ID) [32] or ID verification branch which extracts features of objects”