Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment

By Li Ding, Chenliang Xu. 2018

202 citations recorded by the source corpus.

In this work, we address the task of weakly-supervised human action segmentation in long, untrimmed videos. Recent methods have relied on expensive learning models, such as Recurrent Neural Networks (RNN) and Hidden Markov Models (HMM). However, these methods suffer from expensive computational cost, thus are unable to be deployed in large scale. To overcome the limitations, the keys to our design are efficiency and scalability. We propose a novel action modeling framework, which consists of a new temporal convolutional network, named Temporal Convolutional Feature Pyramid Network (TCFPN), for predicting frame-wise action labels, and a novel training strategy for weakly-supervised sequence modeling, named Iterative Soft Boundary Assignment (ISBA), to align action sequences and update the network in an iterative fashion. The proposed framework is evaluated on two benchmark datasets, Breakfast and Hollywood Extended, with four different evaluation metrics. Extensive experimental results show that our methods achieve competitive or superior performance to state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “79 TCFPN-ISBA [8] 34.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Previous weakly-supervised approaches utilised video-level or transcript supervision, where the set [64,65,66,67,68,69] or sequence [70,71,72,73,74,75] of actions in the video are used in training, without temporal bounds.”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, methodology): “[17] proposed a temporal convolutional feature pyramid network to predict frame-wise labels and use soft 1The language signal should not be treated as supervision since the steps are not directly given, but need to be further explored in an unsupervised manner.”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, methodology): “- TCFPN [7] is also based on action alignment.”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, background): “Temporal models that are built into the architecture [6, 8, 12, 29] are generally favoured because they allow frameworks to be learned end-to-end.”
Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data (2019, methodology): “ors, we regard the task of temporal alignment of frames to a given ordered set of action classes. This task has been introduced by [4] and has so far been used for most weak learning evaluations e.g. [13,18,21,9,22]. For this task, the transcript of a video as well as the video itself is available a”
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding (2020, background): “Besides action recognition, other tasks of action understanding, including action detection and localization [14, 47, 54, 21, 16, 36], action segmentation [26, 9], and action generation [28, 41], also attract many researchers.”
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation (2019, background): “Similarly, Ding and Xu [5] train a temporal convolutional feature pyramid network in an iterative manner starting from a linear alignment.”
Weakly Supervised Energy-Based Learning for Action Segmentation (2019, methodology): “We use the following four standard metrics, as in [1, 7].”
Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization (2019, background): “Researchers have also studied action localization with other types of weak supervision, such as movies scripts [13], ordered action lists [11], and web images [14].”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, background): “Some works [10, 18, 24] assume the transcript includes knowledge of ‘background’, specifying whether the actions occur in succession or with gaps.”