Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition

By Francisco Javier Ordonez, Daniel Roggen. 2016

2,615 citations recorded by the source corpus.

Human activity recognition (HAR) tasks have traditionally been solved using engineered features obtained by heuristic processes. Current research suggests that deep convolutional neural networks are suited to automate feature extraction from raw sensor inputs. However, human activities are made of complex sequences of motor movements, and capturing this temporal dynamics is fundamental for successful HAR. Based on the recent success of recurrent neural networks for time series domains, we propose a generic deep framework for activity recognition based on convolutional and LSTM recurrent units, which: (i) is suitable for multimodal wearable sensors; (ii) can perform sensor fusion naturally; (iii) does not require expert knowledge in designing features; and (iv) explicitly models the temporal dynamics of feature activations. We evaluate our framework on two datasets, one of which has been used in a public activity recognition challenge. Our results show that our framework outperforms competing deep non-recurrent networks on the challenge dataset by 4% on average; outperforming some of the previous reported results by up to 9%. Our results show that the framework can be applied to homogeneous sensor modalities, but can also fuse multimodal sensors to improve performance. We characterise key architectural hyperparameters’ influence on performance to provide insights about their optimisation.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, background): “Ordonez et al. [76] have proposed a DL framework that combines the strength of CNN and LSTM neural networks for human activity recognition from wearable sensor data (accelerometer, gyroscope, etc.).”
Converging blockchain and next-generation artificial intelligence technologies to decentralize and accelerate biomedical research and healthcare (2015, methodology): “For example, RNN model called DeepConvLSTM, a model that combine convolutional networks and recurrent networks with Long Short-Term Memory (LSTM) architecture was applied on recordings from on-body sensors to predict movements and gestures [46].”
Remembering history with convolutional LSTM for anomaly detection (2017, background): “LSTM has shown its success for modeling the motion information for activity recognition [17].”
Materials that couple sensing, actuation, computation, and communication (2015, methodology): “Temporal models, such as dynamic Bayesian networks (including HMMs) and recurrent neural networks (including LSTMs), have been shown to improve classification results for gait recognition [4,94,95,116] and affective touch [56].”
Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network (2018, methodology): “[5] also used bi-directional LSTM in the Named Entity Classification task, while [19,22] stacked multiple convolutional layers to extract hierarchical features in image processing.”
Activity recognition in beach volleyball using a Deep Convolutional Neural Network (2017, background): “Ordóñez and Roggen (2016) used a recurrent network architecture for gesture recognition from IMMU data.”
Multi-head CNN-RNN for multi-time series anomaly detection: An industrial case study (2019, methodology): “As described, time series are processed in a window-based way [29].”
Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum (2016, background): “Finally, significant improvements in human activity recognition based on accelerometers or gyroscopes have been demonstrated [25] by leveraging similar architectures.”
Deep Learning for Sensor-based Activity Recognition: A Survey (2017, methodology): “For multi-modal signals, it is better to use CNN since (Bhattacharya and Lane, 2016) Body-worn, ambiemt RBM Gesture, ADL, transportation Self, D01 (Chen and Xue, 2015) Body-worn CNN ADL Self (Chen et al., 2016b) Body-worn CNN ADL D06 (Edel and K¨oppe, 2016) Body-worn RNN ADL D01, D04, Self (Fang and”
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2021, background): “[207] proposes a hybrid CNN and LSTM model (DeepConvLSTM) for activity recognition using multimodal wearable sensor data.”
Learning a Transferable Change Rule from a Recurrent Neural Network for Land Cover Change Detection (2016, background): “RNN models have gained significant attention for solving many challenging problems involving sequential time series data, especially Long Short-Term Memory (LSTM) models [27].”