Learning Temporal Regularity in Video Sequences

By Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K. Roy-Chowdhury, Larry S. Davis. Computer Vision and Pattern Recognition, 2016

1,362 citations recorded by the source corpus.

Perceiving meaningful activities in a long video sequence is a challenging problem due to ambiguous definition of 'meaningfulness' as well as clutters in the scene. We approach this problem by learning a generative model for regular motion patterns (termed as regularity) using multiple sources with very limited supervision. Specifically, we propose two methods that are built upon the autoencoders for their ability to work with little to no supervision. We first leverage the conventional handcrafted spatio-temporal local features and learn a fully connected autoencoder on them. Second, we build a fully convolutional feed-forward autoencoder to learn both the local features and the classifiers as an end-to-end learning framework. Our model can capture the regularities from multiple datasets. We evaluate our methods in both qualitative and quantitative ways - showing the learned regularity of videos in various aspects and demonstrating competitive performance on anomaly detection datasets as an application.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/97e7c94a78ae17cfb90848c1cfca8c431082a7b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AnomalyNet: An Anomaly Detection Network for Video Surveillance (2019, methodology): “We compare our method with some state-of-the-art approaches including two handcrafted features based methods [7], [66] and six deep learning based methods [3], [5], [14], [34], [37], [39], [64] on these two datasets.”
Latent Space Autoregression for Novelty Detection (2018, methodology): “The awareness of an autonomous system to recognize unknown events enables applications in several domains, ranging from video surveillance [7, 11], to defect detection [19] to medical imaging [35].”
Real-World Anomaly Detection in Surveillance Videos (2018, methodology): “Recently, [18, 40] used deep learning based autoencoders to learn the model of normal behaviors and employed reconstruction loss to detect anomalies.”
Generative Probabilistic Novelty Detection with Adversarial Autoencoders (2018, methodology): “[26, 27] used deep learning based autoencoders to learn the model of normal behaviors and employed a reconstruction loss to detect outliers.”
MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection (2021, methodology): “[33, 16, 7, 17, 15, 5, 36] , which encodes the usual pattern with only normal training samples, and then detects the distinctive encoded patterns as anomalies.”
Learning Memory-Guided Normality for Anomaly Detection (2020, methodology): “, a convolutional AE (Conv-AE) [9], a 3D Conv-AE [50], a recurrent neural network (RNN) [29, 26, 25], and a generative adversarial network (GAN) [33].”
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision (2020, neutral): “[14] 30.”
Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images (2018, background): “Similar to [5,2], we then make the architecture fully convolutional to ensure that spatial information is not lost in the bottleneck of the model.”
Graph Convolutional Label Noise Cleaner: Train a Plug-And-Play Action Classifier for Anomaly Detection (2019, background): “Recently, with the great success of deep learning, a few researchers design deep neural networks on abstraction feature learning [19, 12, 42] or video prediction learning [40] for anomaly detection.”
Autoencoder with recurrent neural networks for video forgery detection (2017, methodology): “It is also worth noting that in the literature autoencoders have been already proposed for anomaly detection in order to solve a one class classification problem [12], and have been applied in very different applications [13, 14].”
A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction (2021, methodology): “We compare our proposed HF2-VAD with state-of-the-art methods, including (1) reconstruction-based methods: Conv-AE [11], ConvLSTMAE [31], GMFC-VAE [7], MemAE [8] and MNADR [38]; (2) prediction-based methods: Frame-Pred.”
Spatio-Temporal AutoEncoder for Video Anomaly Detection (2017, methodology): “Similar to previous works [3, 32], the reconstruction error is expressed as the Euclidean loss:”
Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals (2021, methodology): “Convolutional Autoencoders (CAEs) [21] are an important method of video anomaly detection, which are able of capturing the 2D image structure since the weights are shared among all locations in the input image.”
Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (2020, background): “showed that an autoencoder is able to learn regular dynamics and identify irregularity in long-duration videos [18].”
Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos (2019, methodology): “To understand how the detection of anomalies is made by all models, we visually compare in Figure 4 the map of anomaly scores produced by MPED-RNN to those produced by Conv-AE [13] and Liu et al.”
Detecting anomalous events in videos by learning deep representations of appearance and motion (2017, methodology): “As shown in the table, the proposed framework outperforms [56] in the Ped1, Ped2 and Subway exit datasets.”