Temporal Convolutional Networks for Action Segmentation and Detection

By Colin S. Lea, Michael D. Flynn, René Vidal, Austin Reiter, Gregory Hager. 2016

2,103 citations recorded by the source corpus.

The ability to identify and temporally segment fine-grained human actions throughout a video is crucial for robotics, surveillance, education, and beyond. Typical approaches decouple this problem by first extracting local spatiotemporal features from video frames and then feeding them into a temporal classifier that captures high-level temporal patterns. We describe a class of temporal models, which we call Temporal Convolutional Networks (TCNs), that use a hierarchy of temporal convolutions to perform fine-grained action segmentation or detection. Our Encoder-Decoder TCN uses pooling and upsampling to efficiently capture long-range temporal patterns whereas our Dilated TCN uses dilated convolutions. We show that TCNs are capable of capturing action compositions, segment durations, and long-range dependencies, and are over a magnitude faster to train than competing LSTM-based Recurrent Neural Networks. We apply these models to three challenging fine-grained datasets and show large improvements over the state of the art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, methodology): “The video encoder is a two-layer GRU [5] instead of TCN [24] used in the original paper. and their cross-modal correspondence.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “This definition is closely related to temporal segmentation [80], but segmentation assumes nonoverlapping segments and is thus unsuitable for EPIC-KITCHENS-100.”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Note that our temporally dilated convolution is different from the one used in [13], which designed a speciﬁc layer to combine representations from different frames with various step sizes.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “Beside the online action prediction task, the problem of temporal action detection [22], [56], [56], [57], [58], [59], [60], [61], [62], [63],”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “Most of available datasets require subjects to perform actions in a visible and restricted view to avoid occlusion, and this results in limited view data collection and less 29 SBU Kinect Interaction [149] Depth+ Skeleton PRNN Side Information 89.20% [130] Skeleton Co-occurrence +LSTM None 90.41% [1”
Interpretable 3D Human Action Analysis with Temporal Convolutional Networks (2017, methodology): “A D-dimensional feature vector, whether it is a deep feature from a spatial CNN such as fc7 activation of AlexNet [19] or a set of kinematic features [20], is extracted per each video frame.”
Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation (2018, methodology): “Motivated by the temporal convolutional network (TCN) [23], [24], [25], we propose a fully convolutional separation module that consists of stacked 1-D dilated convolutional blocks, as shown in Figure 1 B.”
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment (2018, methodology): “We also implement the ED-TCN [12] as our baseline method, which has not been implemented on the two datasets yet.”
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning (2018, background): “Purely CNN based approaches such as structured segment networks [37] or temporal convolutional networks [18] have recently shown convincing results on several action segmentation benchmarks.”
WHAMR!: Noisy and Reverberant Single-Channel Speech Separation (2019, methodology): “Interestingly, we observe consistently better performance from the BLSTM model over the TCN model, which is somewhat unexpected.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “Temporal Action segmentation (also referred to as temporal action localization or temporal action detection) consists in detecting all temporal segments containing the action happening in a video [83, 84, 85, 86].”
PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting (2022, methodology): “T IME series forecasting is a research-intensive ﬁeld, especially with the increasing of applying various deep learning frameworks for prediction such as models based on LSTM [1], Temporal Convolutional Network (TCN) [2], and Transformer [3].”
Weakly Supervised Energy-Based Learning for Action Segmentation (2019, methodology): “While a review of fully supervised action segmentation [25, 14, 18, 16] is beyond our scope, it is worth mentioning that our approach uses the same recurrent deep models for frame labeling as in [23, 25, 6].”
Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization (2019, background): “Other works on spatial-temporal action detection [17] and video temporal segmentation [27] are beyond our scope.”
Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification (2018, methodology): “We utilize TCN [10,11] to extract such high-level representation from raw features.”
A Survey of Sound Source Localization with Deep Learning Methods (2021, neutral): “An indirect use of an MLP is explored in [111], where the authors use a 3-layer MLP to enhance the interaural phase difference (IPD) (see Section V) of the input signal, which is then used for DoA estimation.”
Gaussian Temporal Awareness Networks for Action Localization (2019, background): “One fundamental challenge that underlies the success of these advances is action detection in videos from both temporal aspect [6, 9, 17, 30, 39, 43] and spatiotemporal aspect [11, 18].”