RF-based 3D skeletons

By Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, Antonio Torralba. Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, 2018

408 citations recorded by the source corpus.

This paper introduces RF-Pose3D, the first system that infers 3D human skeletons from RF signals. It requires no sensors on the body, and works with multiple people and across walls and occlusions. Further, it generates dynamic skeletons that follow the people as they move, walk or sit. As such, RF-Pose3D provides a significant leap in RF-based sensing and enables new applications in gaming, healthcare, and smart homes. RF-Pose3D is based on a novel convolutional neural network (CNN) architecture that performs high-dimensional convolutions by decomposing them into low-dimensional operations. This property allows the network to efficiently condense the spatio-temporal information in RF signals. The network first zooms in on the individuals in the scene, and crops the RF signals reflected off each person. For each individual, it localizes and tracks their body parts - head, shoulders, arms, wrists, hip, knees, and feet. Our evaluation results show that RF-Pose3D tracks each keypoint on the human body with an average error of 4.2 cm, 4.0 cm, and 4.9 cm along the X, Y, and Z axes respectively. It maintains this accuracy even in the presence of multiple people, and in new environments that it has not seen in the training set. Demo videos are available at our website: http://rfpose3d.csail.mit.edu.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/e808179eecb1251a64dc96c15ad342d0c66199ac

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chen Chen
2. Nasser Kehtarnavaz
3. Taojiannan Yang
4. Sijie Zhu
5. Hamed Haddadi
6. Ce Zheng
7. Ju Shen
8. Ruixu Liu

In Electrical and Electronic Engineering

1. Kun Qian
2. Dinh Thai Hoang
3. Zheng Yang
4. Chenshu Wu
5. Dusist Niyato
6. Yunhao Liu
7. Dong In Kim
8. Guidong Zhang

In Human-Computer Interaction

1. Taojiannan Yang
2. Ju Shen
3. Mubarak Shah
4. Arindam Sengupta
5. Ruixu Liu
6. Siyang Cao
7. Feng Jin
8. Ce Zheng

Citations in the corpus

• mm-Pose: Real-Time Human Skeletal Posture Estimation Using mmWave Radars and CNNs (2019, methodology): “A circular array of vision sensors was used to capture the scene and Open-Pose skeletal data served as the ground truth for supervised training [26].”
• Widar3.0: Zero-Effort Cross-Domain Gesture Recognition With Wi-Fi (2021, background): “Due to complexity of human activity, existing approaches extract signal features, either statistical [10], [31], [33], [34], [35], [36], [37] or physical [11], [13], [16], [38], [39], [40], [41], [42], [43] ones, and map them to discrete activities.”
• Deep Learning-based Human Pose Estimation: A Survey (2020, background): “Later the extended version, named RF-Pose3D [235], can estimate 3D skeletons for multi-person.”
• Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi (2019, background): “Due to complexity of human activity, existing approaches extract signal features, either statistical [14, 15, 23, 28, 30, 45, 49] or physical [6, 31, 34, 38, 39, 44, 51, 52] ones, and map them to discrete activities.”
• Toward Smart Wireless Communications via Intelligent Reflecting Surfaces: A Contemporary Survey (2019, background): “From this view point, the use of the IRS as an array of smart sensors will have rich applications in wireless sensing, e.g., indoor positioning [96], [108] and human pose understanding [201].”
• Deep Learning in Mobile and Wireless Networking: A Survey (2018, methodology): “[257] Estimating 3D human skeleton from radio frequently signal Cloud-based CNN”