Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection

By Tharindu Fernando, Simon Denman, Sridha Sridharan, Clinton Fookes. Neural Networks, 2017

361 citations recorded by the source corpus.

As humans we possess an intuitive ability for navigation which we master through years of practice; however existing approaches to model this trait for diverse tasks including monitoring pedestrian flow and detecting abnormal events have been limited by using a variety of hand-crafted features. Recent research in the area of deep-learning has demonstrated the power of learning features directly from the data; and related research in recurrent neural networks has shown exemplary results in sequence-to-sequence problems such as neural machine translation and neural image caption generation. Motivated by these approaches, we propose a novel method to predict the future motion of a pedestrian given a short history of their, and their neighbours, past behaviour. The novelty of the proposed method is the combined attention model which utilises both "soft attention" as well as "hard-wired" attention in order to map the trajectory information from the local neighbourhood to the future positions of the pedestrian of interest. We illustrate how a simple approximation of attention weights (i.e. hard-wired) can be merged together with soft attention weights in order to make our model applicable for challenging real world scenarios with hundreds of neighbours. The navigational capability of the proposed method is tested on two challenging publicly available surveillance databases where our model outperforms the current-state-of-the-art methods. Additionally, we illustrate how the proposed architecture can be directly applied for the task of abnormal event detection without handcrafting the features.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1939447e7ee0140d0e3580a27bd9d71782bc2e9c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Learning for Spatio-Temporal Data Mining: A Survey (2019, background): “GraphCNN [85], [155], [94], [111], [144], [22], [92], [175], [44], [8], [85], [155] RNN(LSTM,GRU) [42], [77], [165], [99], [91], [163], [35], [159], [64], [38], [135], [181], [88], [81], [190], [37], [169], [166], [41], [65], [192] [126], [27], [177], [90], [23], [89], [178], [17], [179], [101], [97”
Human motion trajectory prediction: a survey (2019, methodology): “2018) or adding a better pooling mechanism with relative importance of each person in the vicinity of the target agent (Xu et al. 2018; Fernando et al. 2018; Pei et al. 2019).”
Social Attention: Modeling Attention in Human Crowds (2017, background): “The authors would also like to point out a very recent work [4] who also consider all agents in the environment, rather than just the local neighborhood, using attention.”
STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction (2019, background): “Many recent works have pointed out that when considering the inﬂuences of other pedestrians, each pedestrian in the scene is necessary [38, 10, 12, 33].”
Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion (2022, methodology): “Some methods apply the attention models [10, 19, 37, 47, 51] to explore the key interactions of the crowd.”
Deep Learning-Based Vehicle Behavior Prediction for Autonomous Driving Applications: A Review (2020, background): “ry). One part of this general problem is to predict the behaviour of pedestrians (or generally speaking, the vulnerable road-users), which is well-studied in computer vision literature [2], [3], [4], [5]. There are also several review papers on pedestrian behaviour prediction such as [6], [7], [8]. ”
Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks (2019, background): “Instead of handcrafting these features, modern approaches rely on recurrent neural networks that learn these parameters directly from the data [10, 16], while incorporating some means of capturing human interaction features [15, 27, 28].”
A Survey on Anomaly Detection for Technical Systems using LSTM Networks (2021, methodology): “This network type is utilized in [32] to detect anomalies based on the cell states being propagated through the network.”
End-to-end Recurrent Multi-Object Tracking and Trajectory Prediction with Relational Reasoning (2019, methodology): “Attention mechanisms have also been employed to query the most relevant information, such as neighbouring pedestrians, in a learnable fashion [24, 5, 23].”
A Survey on Trajectory-Prediction Methods for Autonomous Driving (2022, methodology): “AM is usually used in the trajectory prediction task [113]–[115].”
Crowd-Robot Interaction: Crowd-Aware Robot Navigation With Attention-Based Deep Reinforcement Learning (2018, methodology): “More recently, generative models are used for improved accuracy and efficiency [15], [40], [41].”