A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training

By Runpeng Cui, Hu Liu, Changshui Zhang. IEEE transactions on multimedia, 2019

395 citations recorded by the source corpus.

This work develops a continuous sign language (SL) recognition framework with deep neural networks, which directly transcribes videos of SL sentences to sequences of ordered gloss labels. Previous methods dealing with continuous SL recognition usually employ hidden Markov models with limited capacity to capture the temporal information. In contrast, our proposed architecture adopts deep convolutional neural networks with stacked temporal fusion layers as the feature extraction module, and bidirectional recurrent neural networks as the sequence learning module. We propose an iterative optimization process for our architecture to fully exploit the representation capability of deep neural networks with limited data. We first train the end-to-end recognition model for alignment proposal, and then use the alignment proposal as strong supervisory information to directly tune the feature extraction module. This training process can run iteratively to achieve improvements on the recognition performance. We further contribute by exploring the multimodal fusion of RGB images and optical flow in sign language. Our method is evaluated on two challenging SL recognition benchmarks, and outperforms the state of the art by a relative improvement of more than 15% on both databases.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/f96ffd8c71b97e46eb3ba48263c9012d197494d4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language (2018, methodology): “pochs. In this paper, we focused on RGB only algorithms and did not use optical ﬂow for any of the implementations. It is a proven fact that using optical ﬂow as second stream in train and test stage [17,25,58] or just train stage [1] boosts the performance of prediction. Herein, we describe the met”
Improving Sign Language Translation with Monolingual Data by Sign Back-Translation (2021, positive): “To enable end-to-end training, connectionist temporal clas-siﬁcation (CTC) [15] is widely adopted by continuous SLR models [7, 14, 30, 33, 49].”
Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation (2020, background): “More recent work has tackled continuous data [42, 32, 17, 18], but the move from recognition to translation is still in its infancy [9].”