Generalizing Hand Segmentation in Egocentric Videos With Uncertainty-Guided Model Adaptation

By Minjie Cai, Feng Lu, Yoichi Sato. 2020

47 citations recorded by the source corpus.

Although the performance of hand segmentation in egocentric videos has been significantly improved by using CNNs, it still remains a challenging issue to generalize the trained models to new domains, e.g., unseen environments. In this work, we solve the hand segmentation generalization problem without requiring segmentation labels in the target domain. To this end, we propose a Bayesian CNN-based model adaptation framework for hand segmentation, which introduces and considers two key factors: 1) prediction uncertainty when the model is applied in a new domain and 2) common information about hand shapes shared across domains. Consequently, we propose an iterative self-training method for hand segmentation in the new domain, which is guided by the model uncertainty estimated by a Bayesian CNN. We further use an adversarial component in our framework to utilize shared information about hand shapes to constrain the model adaptation process. Experiments on multiple egocentric datasets show that the proposed method significantly improves the generalization performance of hand segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lingzhi Zhang
2. Dana Rezazadegan
3. Tony R. Martinez
4. Fanqing Lin
5. Vladimir Makarenkov
6. Abbas Khosravi
7. Mohammad Ghavamzadeh
8. Xiaochun Cao

In Computer Vision and Pattern Recognition

1. Paul W. Fieguth
2. Xiaochun Cao
3. Simon Stent
4. Tony R. Martinez
5. Fanqing Lin
6. Shenghao Zhou
7. Lingzhi Zhang
8. Saeid Nahavandi

In Representation & Self-Supervised Learning

1. Tony R. Martinez
2. Fanqing Lin
3. Abbas Khosravi
4. Mohammad Ghavamzadeh
5. Xiaochun Cao
6. Paul W. Fieguth
7. Saeid Nahavandi
8. Shenghao Zhou

In Human-Computer Interaction

1. Tony R. Martinez
2. Fanqing Lin
3. Lingzhi Zhang
4. Shenghao Zhou
5. Simon Stent
6. Jianbo Shi

Citations in the corpus

• Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “More recently, many works [1,73,4,34,36,37,58,59,61,24,50] have applied deep networks for hand or object segmentation.”
• Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detection (2020, methodology): “YHG [10] Real 488 4 Yes 2 480× 360 Ego3DHands [16] Synth 110,000 ∼214,500 1 No 3 960× 540”
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “[423] worked on the hand segmentation generalization issue without using segmentation labels in the target domain.”