Through-Wall Human Pose Estimation Using Radio Signals

By Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, Dina Katabi. 2018

675 citations recorded by the source corpus.

This paper demonstrates accurate human pose estimation through walls and occlusions. We leverage the fact that wireless signals in the WiFi frequencies traverse walls and reflect off the human body. We introduce a deep neural network approach that parses such radio signals to estimate 2D poses. Since humans cannot annotate radio signals, we use state-of-the-art vision model to provide cross-modal supervision. Specifically, during training the system uses synchronized wireless and visual inputs, extracts pose information from the visual stream, and uses it to guide the training process. Once trained, the network uses only the wireless signal for pose estimation. We show that, when tested on visible scenes, the radio-based system is almost as accurate as the vision-based system used to train it. Yet, unlike vision-based pose estimation, the radio-based system can estimate 2D poses through walls despite never trained on such scenarios. Demo videos are available at our website.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “The idea behind the paired samples is to transfer the annotation or label information via pair-wise sample registration and has been widely used for cross-modal applications (Albanie et al. 2018; Zhao et al. 2018; Thoker and Gall 2019).”
The Sound of Pixels (2018, background): “Our work builds off efforts to learn perceptual models that are “self-supervised” by leveraging natural contextual signals in images [10, 22, 33, 38, 24], videos [46, 32, 43, 44, 13, 20], and even radio signals [48].”
Cross-Modal Knowledge Distillation for Action Recognition (2019, methodology): “Cross-modal distillation has been also used for other tasks such as object detection [20], emotion recognition [21], or human pose estimation [22].”
Source-Free Domain Adaptation for Semantic Segmentation (2021, background): “Since then, a variety of KD methods has been presented for model compression [28, 2, 50, 32], domain adaptation [52, 53], and multi-modal learning [51, 14, 10].”
mm-Pose: Real-Time Human Skeletal Posture Estimation Using mmWave Radars and CNNs (2019, positive): “This was soon followed by RF-Pose , proposed by the same research group in 2018, that used RF heat-maps obtained using two antenna arrays, one vertical and the other horizontal [25].”
3D Human Pose Estimation in the Wild by Adversarial Learning (2018, background): “Although DCNNs fit well on these datasets, when being applied on in-the-wild images, where only 2D ground-truth annotations are available (e.g., the MPII human pose dataset [1]), they may have difficulty in terms of generalization ability due to the large domain ar X iv :1 80 3.”
Towards 3D human pose construction using wifi (2020, background): “The later version, RFPose3D [60], extends this framework to achieve 3D skeleton construction.”
HARK Side of Deep Learning - From Grad Student Descent to Automated Machine Learning (2019, background): “Examples include replacing faces and voices in videos [55], detecting people using WiFi signals [56], deciding whose life to risk in an eminent accident [57] and generating fake news [58].”
Widar3.0: Zero-Effort Cross-Domain Gesture Recognition With Wi-Fi (2021, background): “Due to complexity of human activity, existing approaches extract signal features, either statistical [10], [31], [33], [34], [35], [36], [37] or physical [11], [13], [16], [38], [39], [40], [41], [42], [43] ones, and map them to discrete activities.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “Zhao [269] 7 RGB frames Heatmaps One 7 Confidence maps 7”
Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi (2019, background): “Due to complexity of human activity, existing approaches extract signal features, either statistical [14, 15, 23, 28, 30, 45, 49] or physical [6, 31, 34, 38, 39, 44, 51, 52] ones, and map them to discrete activities.”
Fast Human Pose Estimation (2018, background): “A more similar work is the latest radio signals based pose model that also adopts the idea of knowledge distillation [38].”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[232] proposed a RF-based deep learning method, named RFPose, to estimate 2D pose for multi-person scenarios.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, background): “, 2014), radio frequency signal (Zhao et al., 2018), and multi-view inputs (Rhodin et al.”
Elderly Fall Detection Systems: A Literature Survey (2020, methodology): “Li et al. (2018) combined accelerometer data from smartphones and Kinect depth data as well as smartphone camera signals.”