MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation

By Yazan Abu Farha, Juergen Gall. Computer Vision and Pattern Recognition, 2019

881 citations recorded by the source corpus.

Temporally locating and classifying action segments in long untrimmed videos is of particular interest to many applications like surveillance and robotics. While traditional approaches follow a two-step pipeline, by generating frame-wise probabilities and then feeding them to high-level temporal models, recent approaches use temporal convolutions to directly classify the video frames. In this paper, we introduce a multi-stage architecture for the temporal action segmentation task. Each stage features a set of dilated temporal convolutions to generate an initial prediction that is refined by the next one. This architecture is trained using a combination of a classification loss and a proposed smoothing loss that penalizes over-segmentation errors. Extensive evaluation shows the effectiveness of the proposed model in capturing long-range dependencies and recognizing action segments. Our model achieves state-of-the-art results on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b4673e744d0ded47fe6df3b6314f79a41359578b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Temporal Aggregate Representations for Long-Range Video Understanding (2020, background): “Temporal models that are built into the architecture [6, 8, 12, 29] are generally favoured because they allow frameworks to be learned end-to-end.”
MAXIM: Multi-Axis MLP for Image Processing (2022, background): “These multi-stage frameworks are generally inspired by their success on higher-level problems such as pose estimation [17, 47], action segmentation [24, 46], and image generation [113, 114].”
Coherent Temporal Synthesis for Incremental Action Segmentation (2024, neutral): “Fully supervised approaches rely on the dense annotation of the video frames [12, 53].”
Multi-Stage Progressive Image Restoration (2021, background): “In contrast, multi-stage networks are shown to be more effective than their single-stage counterparts in highlevel vision problems such as pose-estimation [14, 46, 54], scene parsing [15] and action segmentation [20, 26, 45].”