DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing

By Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, Tarek F. Abdelzaher. The Web Conference, 2016

709 citations recorded by the source corpus.

Mobile sensing and computing applications usually require time-series inputs from sensors, such as accelerometers, gyroscopes, and magnetometers. Some applications, such as tracking, can use sensed acceleration and rate of rotation to calculate displacement based on physical system models. Other applications, such as activity recognition, extract manually designed features from sensor inputs for classification. Such applications face two challenges. On one hand, on-device sensor measurements are noisy. For many mobile applications, it is hard to find a distribution that exactly describes the noise in practice. Unfortunately, calculating target quantities based on physical system and noise models is only as accurate as the noise assumptions. Similarly, in classification applications, although manually designed features have proven to be effective, it is not always straightforward to find the most robust features to accommodate diverse sensor noise patterns and heterogeneous user behaviors. To this end, we propose DeepSense, a deep learning framework that directly addresses the aforementioned noise and feature customization challenges in a unified manner. DeepSense integrates convolutional and recurrent neural networks to exploit local interactions among similar mobile sensors, merge local interactions of different sensory modalities into global interactions, and extract temporal relationships to model signal dynamics. DeepSense thus provides a general signal estimation and classification framework that accommodates a wide range of applications. We demonstrate the effectiveness of DeepSense using three representative and challenging tasks: car tracking with motion sensors, heterogeneous human activity recognition, and user identification with biometric motion analysis. DeepSense significantly outperforms the state-of-the-art methods for all three tasks. In addition, we show that DeepSense is feasible to implement on smartphones and embedded devices thanks to its moderate energy consumption and low latency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/1b3675fc0f2b16743b1e1f0c2f84829cfdb3d34f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “In DeepSense, most computation-intensive tasks are performed by GPU, and only the padding and memory allocation are performed by the CPU.”
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2021, background): “[315] proposes a light-weight CNN and RNN-based system that accounts for noisy sensor readings from smartphones and automatically learns more local and global features between sensor windows to improve classification and regression performance on smartphones.”
Deep Learning With Edge Computing: A Review (2019, background): “DeepSense [56] is one framework geared toward IoT data fusion leveraging spatiotemporal relationships.”
When Will You Arrive? Estimating Travel Time Based on Deep Neural Networks (2018, background): “To overcome such issue, we use two stacked Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber 1997) layers instead.”
Widar3.0: Zero-Effort Cross-Domain Gesture Recognition With Wi-Fi (2021, methodology): “CNN is a useful technique to extract spatial features and compress data [51], [52], and it is especially suitable for handling the single BVP, which is highly sparse but preserves spatial locality, as a velocity component usually corresponds to the same body part as its neighbors with similar…”
Deep Learning for Sensor-based Human Activity Recognition (2020, background): “rated superior performance to individual networks on three benchmark datasets. Aside from the variants of the RNN structure, some researchers also studied different RNN cells. For example, Yao et al. [169] leveraged the Gated Recurrent Units (GRUs) instead of LSTM cells to construct an RNN and appli”
DeepMove: Predicting Human Mobility with Attentional Recurrent Networks (2018, background): “[39] propose DeepSense: a unified deep learning framework for mobile sensing data.”
Human Activity Recognition using Inertial, Physiological and Environmental Sensors: A Comprehensive Survey (2020, background): “ing [59], house cleaning [4], [59]–[61], driving [62]–[65], smoking [66], swimming [67], or biking [6], [43], [64], [68].”
Deep Learning in Mobile and Wireless Networking: A Survey (2018, neutral): “[292] Car tracking,heterogeneous human activity recognition and user identification Edge-based CNN, RNN”
End-to-End Learning From Spectrum Data: A Deep Learning Approach for Wireless Signal Identification in Spectrum Monitoring Applications (2017, background): “The authors of [15] propose a uniﬁed deep learning framework for mobile sensing data.”
Deep Learning for Sensor-based Activity Recognition: A Survey (2017, methodology): “For multi-modal signals, it is better to use CNN since (Bhattacharya and Lane, 2016) Body-worn, ambiemt RBM Gesture, ADL, transportation Self, D01 (Chen and Xue, 2015) Body-worn CNN ADL Self (Chen et al., 2016b) Body-worn CNN ADL D06 (Edel and K¨oppe, 2016) Body-worn RNN ADL D01, D04, Self (Fang and”
Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi (2019, methodology): “Note that for recognition systems which involve more sophisticated activities with longer durations, the GRU-based models can be transformed into more complex versions [11, 47].”