Recurrent Mixture Density Network for Spatiotemporal Visual Attention

By Loris Bazzani, Hugo Larochelle, Lorenzo Torresani. 2016

142 citations recorded by the source corpus.

In many computer vision tasks, the relevant information to solve the problem at hand is mixed to irrelevant, distracting information. This has motivated researchers to design attentional models that can dynamically focus on parts of images or videos that are salient, e.g., by down-weighting irrelevant pixels. In this work, we propose a spatiotemporal attentional model that learns where to look in a video directly from human fixation data. We model visual attention with a mixture of Gaussians at each frame. This distribution is used to express the probability of saliency for each pixel. Time consistency in videos is modeled hierarchically by: 1) deep 3D convolutional features to represent spatial and short-term time relations and 2) a long short-term memory network on top that aggregates the clip-level representation of sequential clips and therefore expands the temporal domain from few frames to seconds. The parameters of the proposed model are optimized via maximum likelihood estimation using human fixations as training data, without knowledge of the action in each video. Our experiments on Hollywood2 show state-of-the-art performance on saliency prediction for video. We also show that our attentional model trained on Hollywood2 generalizes well to UCF101 and it can be leveraged to improve action classification accuracy on both datasets.

Subfields: Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “Firstly, MoE has usually been instantiated as mixture of Gaussians to model data in continuous domains (Jacobs et al., 1991; Graves, 2013; Bazzani et al., 2016).”
Pose-conditioned Spatio-Temporal Attention for Human Action Recognition (2017, background): “Bazzani et al [6] learn spatial saliency maps represented by mixtures of Gaussians, whose parameters are included into the internal state of a LSTM network.”
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data (2016, background): “Many applications leverage predicted saliency maps for performance enhancement (Yu, Mann, and Gosine 2010; Jiang, Xu, and Zhao 2014; Bazzani, Larochelle, and Torresani 2016), which explicitly learn the saliency maps guided by human labeled groundtruths.”
Human Action Recognition: Pose-Based Attention Draws Focus to Hands (2017, background): “Bazzani et al [6] learn spatial saliency maps represented by mixtures of Gaussians, whose parameters are included into the internal state of a LSTM network.”
Simultaneous Enhancement and Super-Resolution of Underwater Imagery for Improved Visual Perception (2020, background): “Automatic saliency prediction over a sequence of frames is also explored extensively [6] because spatio-temporal features capture information about the motion and interaction among objects in a scene, which are important cues for attention modeling.”
Using Human Gaze to Improve Robustness Against Irrelevant Objects in Robot Manipulation Tasks (2020, background): “Many recent studies have investigated predicting the saliency map [28]–[33].”
Semantic Segmentation of Underwater Imagery: Dataset and Benchmark (2020, background): “Moreover, this semantic knowledge is potentially useful for learning spatio-temporal attention modeling and visual question answering [47], [48], i.”
Predicting the Driver's Focus of Attention: The DR(eye)VE Project (2017, methodology): “From left to right: input clip, ground truth map, our prediction, prediction of the previous version of the model [43], prediction of RMDN [3] and prediction of MLNet [12].”