XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

By Ho Kei Cheng, Alexander Gerhard Schwing. European Conference on Computer Vision, 2022

669 citations recorded by the source corpus.

We present XMem, a video object segmentation architecture for long videos with unified feature memory stores inspired by the Atkinson-Shiffrin memory model. Prior work on video object segmentation typically only uses one type of feature memory. For videos longer than a minute, a single feature memory model tightly links memory consumption and accuracy. In contrast, following the Atkinson-Shiffrin model, we develop an architecture that incorporates multiple independent yet deeply-connected feature memory stores: a rapidly updated sensory memory, a high-resolution working memory, and a compact thus sustained long-term memory. Crucially, we develop a memory potentiation algorithm that routinely consolidates actively used working memory elements into the long-term memory, which avoids memory explosion and minimizes performance decay for long-term prediction. Combined with a new memory reading mechanism, XMem greatly exceeds state-of-the-art performance on long-video datasets while being on par with state-of-the-art methods (that do not work on long videos) on short-video datasets. Code is available at https://hkchengrex.github.io/XMem

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/01724c36660359545e1368fc80c99f4bde44a190

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Segment Anything in High Quality (2023, methodology): “We adopt the SOTA model XMem [7] as our video boxes prompt generator while reusing its object association prediction.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “There are some prior works exploring memory models [59] in various vision tasks in videos, such as video object segmentation (VOS) [18,30,55,56], multi-object tracking (MOT) [2,14,29,73], visual object tracking (VOT) [38, 42, 77, 84], and action understanding [68].”
Foundation models in robotics: Applications, challenges, and the future (2023, positive): “The Track Anything Model (TAM) [62] combines SAM and XMem [165], an advanced video object segmentation (VOS) model, to achieve interactive video object tracking and segmentation.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “XMem [340] proposes a mixed memory design to handle the long video inputs.”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, positive): “Given an object/part query from LLMs, we first invoke open-vocab detector OWL-ViT [15] to obtain a bounding box, feed it into Segment Anything [109] to obtain a mask, and track the mask using video tracker XMEM [110].”