Temporal Convolutional Networks: A Unified Approach to Action Segmentation

By Colin S. Lea, René Vidal, Austin Reiter, Gregory Hager. 2016

1,028 citations recorded by the source corpus.

The dominant paradigm for video-based action segmentation is composed of two steps: first, compute low-level features for each frame using Dense Trajectories or a Convolutional Neural Network to encode local spatiotemporal information, and second, input these features into a classifier such as a Recurrent Neural Network (RNN) that captures high-level temporal relationships. While often effective, this decoupling requires specifying two separate models, each with their own complexities, and prevents capturing more nuanced long-range spatiotemporal relationships. We propose a unified approach, as demonstrated by our Temporal Convolutional Network (TCN), that hierarchically captures relationships at low-, intermediate-, and high-level time-scales. Our model achieves superior or competitive performance using video or sensor data on three public action segmentation datasets and can be trained in a fraction of the time it takes to train an RNN.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, background): “This has led to the development of action segmentation approaches [23, 34, 39] that aim to label every frame in the video and not just to classify trimmed video clips.”
Modeling Long-horizon Tasks as Sequential Interaction Landscapes (2020, background): “[34] introduce a unified approach for action recognition based on hierarchical relationships at different time-scales, while Ahsan et al.”
Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification (2018, methodology): “All results of prior works are excerpted from [11].”
Exploring the Best Loss Function for DNN-Based Low-latency Speech Enhancement with Temporal Convolutional Networks (2020, methodology): “Moreover, architectures using a temporal convolutional network (TCN) [16–18] such as Conv-TasNet are considered to be more suitable than recurrent neural network architectures for real-time implementation as they are unaffected by the start point and can be operated efficiently [15, 18].”
Divide and Conquer: A Deep CASA Approach to Talker-Independent Monaural Speaker Separation (2019, methodology): “performance in various tasks [2], [23], [29], [31].”
LSTM Fully Convolutional Networks for Time Series Classification (2017, background): “[22], let Xt ∈ RF0 be the input feature vector of length F0 for time step t for 0 < t ≤ T .”
U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging (2019, methodology): “It is inspired by the popular U-Net architecture originally proposed for image segmentation [Ronneberger et al., 2015, Koch et al., 2019b, Perslev et al., 2019] and so-called temporal convolutional networks [Lea et al., 2016].”
Single Shot Temporal Action Detection (2017, background): “[19] proposes Temporal Convolutional Networks (TCN) for temporal action segmentation.”
Learning a Generative Model for Multi‐Step Human‐Object Interactions from Videos (2019, methodology): “To recognize and temporally segment actions in videos, we employ an LSTM-based pipeline similar to Lea et al. [LVRH16].”
A decoder-only foundation model for time-series forecasting (2023, methodology): “We report performance of several baselines implemented in the Darts package like TCN (Lea et al., 2016), N-HiTS (Challu et al., 2023) and N-BEATS (Oreshkin et al., 2019).”
Dynamical Variational Autoencoders: A Comprehensive Review (2020, background): “In particular, temporal convolutional networks (TCNs) (Lea et al., 2016) which are based on causal and dilated convolutions have shown to be competitive with RNNs on several sequence modeling tasks, including generative modeling (Aksan and Hilliges, 2019).”
EEG-Based Spatio–Temporal Convolutional Neural Network for Driver Fatigue Evaluation (2019, neutral): “efficiency on time series classification tasks [32].”
Large Language Models Are Zero-Shot Time Series Forecasters (2023, methodology): “For Darts, we use several meth-ods that are implemented directly in the package, including neural network models (TCN [28], N-BEATS [37], N-HiTS [11]) and simple moving average models (ARIMA [8]).”
DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation (2017, methodology): “They have also been used by the instance-sensitive FCN [31] and applied to action segmentation and detection from video frames [32].”
FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks (2019, methodology): “[18] introduce TCN [14, 2] to do this task.”
Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks (2017, methodology): “CNN models have also been used in action/activity recognition [13, 20, 31], for the extraction of shift-invariant local patterns from input sequences as the features of classiﬁcation models.”
Multivariate LSTM-FCNs for Time Series Classification (2018, background): “[25] defines Xt ∈ R0 to be the input feature vector of length F0 for time step t.”