Device-free gesture tracking using acoustic signals

By Wen Wang, Alex X. Liu, Ke Sun. ACM/IEEE International Conference on Mobile Computing and Networking, 2016

434 citations recorded by the source corpus.

Device-free gesture tracking is an enabling HCI mechanism for small wearable devices because fingers are too big to control the GUI elements on such small screens, and it is also an important HCI mechanism for medium-to-large size mobile devices because it allows users to provide input without blocking screen view. In this paper, we propose LLAP, a device-free gesture tracking scheme that can be deployed on existing mobile devices as software, without any hardware modification. We use speakers and microphones that already exist on most mobile devices to perform device-free tracking of a hand/finger. The key idea is to use acoustic phase to get fine-grained movement direction and movement distance measurements. LLAP first extracts the sound signal reflected by the moving hand/finger after removing the background sound signals that are relatively consistent over time. LLAP then measures the phase changes of the sound signals caused by hand/finger movements and then converts the phase changes into the distance of the movement. We implemented and evaluated LLAP using commercial-off-the-shelf mobile phones. For 1-D hand movement and 2-D drawing in the air, LLAP has a tracking accuracy of 3.5 mm and 4.6 mm, respectively. Using gesture traces tracked by LLAP, we can recognize the characters and short words drawn in the air with an accuracy of 92.3% and 91.2%, respectively.

Subfields: Cognitive Neuroscience, Electrical and Electronic Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/476d1877acb919fa79cb917de6309989074ad36a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

1. Kyle Jamieson
2. Jie Xiong
3. Mo Li
4. Yaxiong Xie

Citations in the corpus

• mD-Track: Leveraging Multi-Dimensionality for Passive Indoor Wi-Fi Tracking (2018, background): “Acoustic signal based solutions [35, 40, 54] can only work within a few decimeters.”
• Towards Environment Independent Device Free Human Activity Recognition (2018, methodology): “As suggested in [51], we can extract the phase information through demodulating the received signal.”
• Digital Twin: Values, Challenges and Enablers From a Modeling Perspective (2019, neutral): “[538] W.”