Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables

By Nils Y. Hammerla, Shane Halloran, Thomas Plötz. 2016

996 citations recorded by the source corpus.

Human activity recognition (HAR) in ubiquitous computing is beginning to adopt deep learning to substitute for well-established analysis techniques that rely on hand-crafted feature extraction and classification techniques. From these isolated applications of custom deep architectures it is, however, difficult to gain an overview of their suitability for problems ranging from the recognition of manipulative gestures to the segmentation and identification of physical activities like running or ascending stairs. In this paper we rigorously explore deep, convolutional, and recurrent approaches across three representative datasets that contain movement data captured with wearable sensors. We describe how to train recurrent approaches in this setting, introduce a novel regularisation approach, and illustrate how they outperform the state-of-the-art on a large benchmark dataset. Across thousands of recognition experiments with randomly sampled model configurations we investigate the suitability of each model for different tasks in HAR, explore the impact of hyperparameters using the fANOVA framework, and provide guidelines for the practitioner who wants to apply deep learning in their problem setting.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

IoT Wearable Sensor and Deep Learning: An Integrated Approach for Personalized Human Activity Recognition in a Smart Home Environment (2019, methodology): “Instead, in this paper [31], a deep learning technique based on long short-term memory (LSTM), or recurrent neural network (RNN), is introduced to perform HAR task from data captured with wearable sensors in ubiquitous computing.”
Recent trends in machine learning for human activity recognition—A survey (2018, methodology): “…knowledge of the researcher and the performance of the machine learning techniques is highly dependent on the data representation (Bengio, 2013 Hammerla, Halloran, and Ploetz (2016) have explored DNN, CNN, and RNN for activity dataset and concluded that the RNN performed better than the…”
Sensor-based and vision-based human activity recognition: A comprehensive survey (2020, methodology): “As a result, Hammerla et al. introduced a 1-second sliding window with a 50% overlap to create a HAR dataset that contained approximately 650,0 0 0 samples using wearable sensors [139] .”
Comparison of Data Preprocessing Approaches for Applying Deep Learning to Human Activity Recognition in the Context of Industry 4.0 (2018, methodology): “The convolutional neural network (CNN) is one of the most important deep learning approaches that has been used to perform HAR, and has produced satisfying results in a number of studies [25].”
Multi-task Self-Supervised Learning for Human Activity Detection (2019, background): “Deep learning has been increasingly used for end-to-end HAR with far superior performance that can be achieved through traditional machine learning methods [20, 44, 56, 61, 72].”
Deep Learning for Sensor-based Human Activity Recognition (2020, background): “convolutional neural networks (CNNs) are competent in capturing the local connections of multimodal sensory data, and the translational invariance introduced by locality leads to accurate recognition [57]. Recurrent neural networks (RNNs) extract the temporal dependencies and incrementally learn inf”
Human activity recognition from accelerometer data using Convolutional Neural Network (2017, methodology): “In comparison to the existing approaches which employ multiple, varied sensor data (accelerometer, gyroscope, etc.) [4] [6] [7] [8], our method uses only the triaxial accelerometer data (x, y, and z component signals) collected from a single accelerometer sensor.”
Deep PPG: Large-Scale Heart Rate Estimation with Convolutional Neural Networks (2019, methodology): “[25] compared CNNs to recurrent neural networks (RNN), concluding that CNNs are more suitable for continuous periodic activities.”
Deep learning for healthcare: review, opportunities and challenges (2018, positive): “[75] evaluated CNNs and RNNs with LSTM to predict the freezing of gait in Parkinson disease (PD) patients.”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “For the human activity recognition dataset, we follow the pre-processing procedure described in (Hammerla et al., 2016).”
LSTM Networks Using Smartphone Data for Sensor-Based Human Activity Recognition in Smart Homes (2021, background): “[33] compared different deep neural networks in HAR, including CNN and LSTM, and significantly improved the performance and robustness of recognition.”
Multi-input CNN-GRU based human activity recognition using wearable sensors (2021, methodology): “The proposed model is compared with the models presented in [17, 20–22, 36, 37, 49–52].”
Human Activity Recognition using Inertial, Physiological and Environmental Sensors: A Comprehensive Survey (2020, methodology): “The second most used model was the Long Short-TermMemory (LSTM) model, which was used in 17 papers [7], [83], [89], [102], [107], [112], [125], [130], [137], [139], [152], [153], [172],”