Video-based Sign Language Recognition without Temporal Segmentation

By Jie Huang, Wen-gang Zhou, Qilin Zhang, Houqiang Li, Weiping Li. AAAI Conference on Artificial Intelligence, 2018

482 citations recorded by the source corpus.

Millions of hearing impaired people around the world routinely use some variants of sign languages to communicate, thus the automatic translation of a sign language is meaningful and important. Currently, there are two sub-problems in Sign Language Recognition (SLR), i.e., isolated SLR that recognizes word by word and continuous SLR that translates entire sentences. Existing continuous SLR methods typically utilize isolated SLRs as building blocks, with an extra layer of preprocessing (temporal segmentation) and another layer of post-processing (sentence synthesis). Unfortunately, temporal segmentation itself is non-trivial and inevitably propagates errors into subsequent steps. Worse still, isolated SLR methods typically require strenuous labeling of each word separately in a sentence, severely limiting the amount of attainable training data. To address these challenges, we propose a novel continuous sign recognition framework, the Hierarchical Attention Network with Latent Space (LS-HAN), which eliminates the preprocessing of temporal segmentation. The proposed LS-HAN consists of three components: a two-stream Convolutional Neural Network (CNN) for video feature representation generation, a Latent Space (LS) for semantic gap bridging, and a Hierarchical Attention Network (HAN) for latent space based recognition. Experiments are carried out on two large scale datasets. Experimental results demonstrate the effectiveness of the proposed framework.

Subfields: Biomedical Engineering, Developmental and Educational Psychology, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/a274fcf985aaec1f009cfe4b4042733321a5a8b4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Developmental and Educational Psychology

In Human-Computer Interaction

Citations in the corpus

How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language (2020, background): “Video-Based CSL [16] CSL 178 100 50 7 X 7 X X 7”
Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation (2020, background): “More recent work has tackled continuous data [42, 32, 17, 18], but the move from recognition to translation is still in its infancy [9].”
MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language (2018, background): “Purdue ASL [74] 104 14 no 1834 no Video-Based CSL [33] 178 50 yes 25000 no Signum [71] 465 25 yes 15075 no RWTH-Boston [76] 483 5 no 2207 no RWTH-Phoenix [29] 1080 9 no 6841 yes Devisign [12] 2000 8 no 24000 no”
Improving Sign Language Translation with Monolingual Data by Sign Back-Translation (2021, unknown): “Due to the signiﬁcant social impact and the cross-modality challenge, sign language understanding has been attracting more and more research attention [1, 4, 10, 17, 22, 26, 37, 48].”