D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
By Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, Juan Carlos Niebles. 2019
177 citations recorded by the source corpus.
We address weakly supervised action alignment and segmentation in videos, where only the order of occurring actions is available during training. We propose Discriminative Differentiable Dynamic Time Warping (D3TW), the first discriminative model using weak ordering supervision. The key technical challenge for discriminative modeling with weak supervision is that the loss function of the ordering supervision is usually formulated using dynamic programming and is thus not differentiable. We address this challenge with a continuous relaxation of the min-operator in dynamic programming and extend the alignment loss to be differentiable. The proposed D3TW innovatively solves sequence alignment with discriminative modeling and end-to-end training, which substantially improves the performance in weakly supervised action alignment and segmentation tasks. We show that our model is able to bypass the degenerated sequence problem usually encountered in previous work and outperform the current state-of-the-art across three evaluation metrics in two challenging datasets.
Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Kristen Grauman
- Luc Van Gool
- Jitendra Malik
- Cordelia Schmid
- Trevor Darrell
- Li Fei-Fei
- Andrew Zisserman
- Dacheng Tao
In Control and Systems Engineering
- Dima Damen
- Jonathan Munro
- Sinisa Todorovic
- Li Fei-Fei
- Danfei Xu
- Giovanni Maria Farinella
- Antonino Furnari
- Juan Carlos Niebles
Citations in the corpus
- MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos+ASR to build weakly-supervised object detectors [75], action detectors/classifiers [108, 5, 51, 72], instruction aligners [65, 4, 15], video captioners [85, 38, 74, 89], and visual reference resolvers [41].”
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “Previous weakly-supervised approaches utilised video-level or transcript supervision, where the set [64,65,66,67,68,69] or sequence [70,71,72,73,74,75] of actions in the video are used in training, without temporal bounds.”
- Procedure Planning in Instructional Videos (2019, background): “, such as action recognition and temporal action segmentation [4, 13, 28], state understanding [2], video summarization/captioning [29, 32, 37], retrieval [32] etc.”
- Few-Shot Video Classification via Temporal Alignment (2019, methodology): “Our work is closely related to recent progress on using continuous relaxation of discrete operations to tackle sequence alignment problem [5] and hence allow us to train our entire model end-to-end.”
- Weakly Supervised Energy-Based Learning for Action Segmentation (2019, methodology): “Contributions: Prior work shows that a statistical language model is useful for weakly supervised learning and modeling of video sequences [17, 9, 19, 22, 3].”