DynamoNet: Dynamic Action and Motion Network

By Ali Diba, Vivek Sharma, Luc Van Gool, Rainer Stiefelhagen. 2019

124 citations recorded by the source corpus.

In this paper, we are interested in self-supervised learning the motion cues in videos using dynamic motion filters for a better motion representation to finally boost human action recognition in particular. Thus far, the vision community has focused on spatio-temporal approaches using standard filters, rather we here propose dynamic filters that adaptively learn the video-specific internal motion representation by predicting the short-term future frames. We name this new motion representation, as dynamic motion representation (DMR) and is embedded inside of 3D convolutional network as a new layer, which captures the visual appearance and motion dynamics throughout entire video clip via end-to-end network learning. Simultaneously, we utilize these motion representation to enrich video classification. We have designed the frame prediction task as an auxiliary task to empower the classification problem. With these overall objectives, to this end, we introduce a novel unified spatio-temporal 3D-CNN architecture (DynamoNet) that jointly optimizes the video classification and learning motion representation by predicting future frames as a multi-task learning problem. We conduct experiments on challenging human action datasets: Kinetics 400, UCF101, HMDB51. The experiments using the proposed DynamoNet show promising results on all the datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “Following [47], all these recent works [46, 48, 49] utilize the same concept to learn a steerable filter [46], weather prediction filter [48], an image enhancement filter [49], and a dynamic motion motion representation filter [50] using source-target image pairs.”
Spatiotemporal Contrastive Video Representation Learning (2020, neutral): “2 DynamoNet‡ [12] YT8M(13y) V 88.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “…challenge of self-supervised video representation learning through the lens of video pace prediction. b) Motions of objects in videos: Diba et al. [215] focused on SSL of motions in videos by employing dynamic motion ﬁlters to enhance motion representations, particularly for improving human…”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “Unsupervised learning in videos has followed a similar trajectory with earlier methods focusing on predictive tasks based on motion, color and spatiotemporal ordering [29,43,1, 44,78,85,60,84,58,57,21,51,86,66,22,48,91,16,87,70,45], and contrastive objectives with visual [74, 79, 34, 53, 28, 92] and”
Masked Autoencoders As Spatiotemporal Learners (2022, background): “Related topics include temporal coherence (‘slowness’) [79, 25], future prediction [61, 72, 70, 45, 44, 71, 16], object motion [1, 75, 49, 76], temporal ordering [46, 23, 38, 78, 81], spatiotemporal contrast [58, 62, 30, 22, 51, 56], etc .”
Explaining Motion Relevance for Activity Recognition in Video Deep Learning Models (2020, methodology): “This lack of motion understanding in 3D CNNs was addressed in works such as D3D and DynamoNET [1, 7], which implement additional steps specifically geared towards distilling motion representation in the model during the training phase, and in SlowFast Networks, which experiment with various spatial ”
Omni-Dimensional Dynamic Convolution (2022, methodology): “DynamoNet (Diba et al., 2019) uses dynamically generated motion filters to handle the action recognition problem.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, background): “(Diba et al., 2019) learn motion dynamic of videos with a self-supervised task for video understanding.”
Self-supervised Co-training for Video Representation Learning (2020, unknown): “Although only visual information is used, some approaches exploit a larger training set, e.g. CBT and DynamoNet.”
Learning Spatiotemporal Features via Video and Text Pair Discrimination (2020, methodology): “On the other hand, videos contain richer structure with abundant side information such as motion [7, 37], audio [1, 28], and text [35, 49].”