Hands Holding Clues for Object Recognition in Teachable Machines

By Kyungjun Lee, Hernisa Kacorri. 2019

49 citations recorded by the source corpus.

Camera manipulation confounds the use of object recognition applications by blind people. This is exacerbated when photos from this population are also used to train models, as with teachable machines, where out-of-frame or partially included objects against cluttered backgrounds degrade performance. Leveraging prior evidence on the ability of blind people to coordinate hand movements using proprioception, we propose a deep learning system that jointly models hand segmentation and object localization for object classification. We investigate the utility of hands as a natural interface for including and indicating the object of interest in the camera frame. We confirm the potential of this approach by analyzing existing datasets from people with visual impairments for object recognition. With a new publicly available egocentric dataset and an extensive error analysis, we provide insights into this approach in the context of teachable recognizers.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “Omniglot [23] miniImageNet [49] Meta-Dataset [48] TEgO [24] ORBIT Benchmark”
Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors (2021, methodology): “To manage some of these issues, researchers and blind photographers have adopted strategies including: 1) guiding blind and low vision users through audio feedback so that objects stay in frame [4, 37], 2) estimating the general location of the target, positioning the camera close to the target and ”
What Do We Mean by “Accessibility Research”?: A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019 (2021, background): “4% (N=7) of papers [12,42,66,90,150,160,162]—two of which received no other contribution type code: Flores and Manduchi’s WeAllWalk dataset [42] and Wolter et al.”