Deep Temporal Linear Encoding Networks

By Ali Diba, Vivek Sharma, Luc Van Gool. 2016

240 citations recorded by the source corpus.

The CNN-encoding of features from entire videos for the representation of human actions has rarely been addressed. Instead, CNN work has focused on approaches to fuse spatial and temporal networks, but these were typically limited to processing shorter sequences. We present a new video representation, called temporal linear encoding (TLE) and embedded inside of CNNs as a new layer, which captures the appearance and motion throughout entire videos. It encodes this aggregated information into a robust video feature representation, via end-to-end learning. Advantages of TLEs are: (a) they encode the entire video into a compact feature representation, learning the semantics and a discriminative feature space, (b) they are applicable to all kinds of networks like 2D and 3D CNNs for video classification, and (c) they model feature interactions in a more expressive way and without loss of information. We conduct experiments on two challenging human action datasets: HMDB51 and UCF101. The experiments show that TLE outperforms current state-of-the-art methods on both datasets.

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

SplitNN-driven Vertical Partitioning (2020, methodology): “One can readily employ other encoding methods like Compact Bilinear Pooling (Diba et al. , 2017; Gao et al. , 2016), Temporal Compact Bilinear Pooling (Sharma et al. , 2020), NetVLAD (Arandjelovi ´ c et al. , 2016) instead of the pooling mechanisms for a more robust representation learning.”
Spatio-Temporal Channel Correlation Networks for Action Classification (2018, methodology): “…frames are modeled via LSTMs/RNNs to capture long-term temporal dependencies [13,3], or via feature aggregation and encoding using Bilinear models [1], VLAD [14], Fisher encoding [15] etc. Recently, several temporal architectures have been proposed for video classiﬁcation, where the input to the…”
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition (2017, background): “been proposed by considering video action recognition as a classification task [5, 43, 23, 50, 11, 10, 9, 41, 42, 33, 29].”
3D CNNs with Adaptive Temporal Feature Resolutions (2020, unknown): “In recent years, there has been a tremendous progress for video processing in the light of new and complex deep learning architectures, which are based on variants of 3D CNNs [24, 9, 7, 4, 6, 12, 8].”
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition (2018, methodology): “In LRCN (Donahue et al. 2015), CNN features of video frames are fed into LSTM network for action classiﬁcation.”
DistInit: Learning Video Representations Without a Single Labeled Video (2019, methodology): “Finally, prior works have also shown improvements using the 2D image-based networks (‘teachers’, in our context) directly, as additional features for action recognition [6, 14], hence reinforcing our observation that scene/object information is a very useful cue for video understanding.”
Learning Spatio-Temporal Representation With Local and Global Diffusion (2019, methodology): “This two-stream architecture is further extended by exploiting convolutional fusion [5], spatio-temporal attention [24], temporal segment networks [41, 42] and convolutional encoding [4, 27] for video representation learning.”
Deep Multimodal Feature Encoding for Video Ordering (2020, methodology): “[6], we employ TSN [61] architecture with 3 segments pushed through the C3D ConvNet.”
Multi-teacher Knowledge Distillation for Compressed Video Action Recognition on Deep Neural Networks (2019, background): “For two-stream-based approaches [1, 3, 4, 5, 6], the input to the spatial and temporal streams is RGB frames and stacks of multiple-frame dense optical flow fields, respectively.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[52] aggregated the features of the segments by element-wise multiplication.”
Human Action Recognition and Prediction: A Survey (2018, background): “The network named temporal linear encoding (Diba et al., 2017) aggregates temporal features sampled from a video, and then projects onto a lowdimensional feature space.”
PoTion: Pose MoTion Representation for Action Recognition (2018, unknown): “[7] linearly aggregate CNN features trained for action classification over an entire video clip.”
Holistic Large Scale Video Understanding (2019, methodology): “These methods operate on 2D (individual image-level) [9, 10, 16, 38, 39, 48, 51] or 3D (video-clips or snippets of K frames) [12, 40, 41, 42].”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “Generally speaking, they can be divided into two directions, namely, (1) two-stage solutions [8, 1, 16, 29, 35] which extract spatial feature vectors from video frames and then integrate the obtained local feature sequence into one compact video descriptor for recognition; (2) the 2D [45, 36, 10, 7]”
Compressed Video Action Recognition (2017, background): “Karpathy et al. [14] and TLE [5] consider more complicated fusions and pooling.”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, methodology): “Recent approaches [16], [35], [36] evolve to end-to-end learning and are currently the best at incorporating global temporal information.”
DynamoNet: Dynamic Action and Motion Network (2019, methodology): “Human action recognition [3, 6, 7, 9, 55] in videos has attracted a huge attention over the last decade, due to the potential applications in video surveillance, understanding, analysis, retrieval tasks and more.”
Dynamic Inference: A New Approach Toward Efficient Video Action Recognition (2020, background): “Recently, significant progress has been achieved in terms of the recognition performance in this area following the deep convolutional network paradigm [30, 5, 4, 6, 26, 1, 8, 21].”