Deep ConvLSTM With Self-Attention for Human Activity Decoding Using Wearable Sensors

By Satya Prakash Singh, Madan Kumar Sharma, Aimé Lay-Ekuakille, Deepak Gangwar, Sukrit Gupta. 2020

156 citations recorded by the source corpus.

Decoding human activity accurately from wearable sensors can aid in applications related to healthcare and context awareness. The present approaches in this domain use recurrent and/or convolutional models to capture the spatio-temporal features from time-series data from multiple sensors. We propose a deep neural network architecture that not only captures the spatio-temporal features of multiple sensor time-series data but also selects, learns important time points by utilizing a self-attention mechanism. We show the validity of the proposed approach across different data sampling strategies on six public datasets and demonstrate that the self-attention mechanism gave a significant improvement in performance over deep networks using a combination of recurrent and convolution networks. We also show that the proposed approach gave a statistically significant performance enhancement over previous state-of-the-art methods for the tested datasets. The proposed methods open avenues for better decoding of human activity from multiple body sensors over extended periods of time. The code implementation for the proposed model is available at https://github.com/isukrit/encodingHumanActivity

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Thinagaran Perumal
2. Nicholas Donald Lane
3. Xinchi Qiu
4. Akhil Mathur
5. Taner Topal
6. Daniel J. Beutel
7. Titouan Parcollet
8. Yoshua Bengio

In Computer Vision and Pattern Recognition

1. Thinagaran Perumal
2. Nicholas Donald Lane
3. S. Padmavathi
4. Titouan Parcollet
5. Elangovan Ramanujam

In Signal Processing

1. Nicholas Donald Lane
2. Xinchi Qiu
3. Akhil Mathur
4. Taner Topal
5. Daniel J. Beutel
6. Titouan Parcollet
7. Yoshua Bengio
8. Mirco Ravanelli

Citations in the corpus

• Flower: A Friendly Federated Learning Research Framework (2020, methodology): “Figure 5 shows the system metrics associated with training a DeepConvLSTM (Singh et al., 2021) model for a human activity recognition task on Python-enabled Jetson and Raspberry Pi devices.”
• Human Activity Recognition With Smartphone and Wearable Sensors Using Deep Learning Techniques: A Review (2021, methodology): “Though CNN, LSTM and hybrid model capture Spatio-temporal features, Singh et al. [64] have proposed a self-attention layer to focus on specific information from the embeddings generated by any deep learning model.”