Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos

By Oscar Koller, Necati Cihan Camgöz, Hermann Ney, Richard Bowden. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

365 citations recorded by the source corpus.

In this work we present a new approach to the field of weakly supervised learning in the video domain. Our method is relevant to sequence learning problems which can be split up into sub-problems that occur in parallel. Here, we experiment with sign language data. The approach exploits sequence constraints within each independent stream and combines them by explicitly imposing synchronisation points to make use of parallelism that all sub-problems share. We do this with multi-stream HMMs while adding intermediate synchronisation constraints among the streams. We embed powerful CNN-LSTM models in each HMM stream following the hybrid approach. This allows the discovery of attributes which on their own lack sufficient discriminative power to be identified. We apply the approach to the domain of sign language recognition exploiting the sequential parallelism to learn sign language, mouth shape and hand shape classifiers. We evaluate the classifiers on three publicly available benchmark data sets featuring challenging real-life sign language with over 1,000 classes, full sentence based lip-reading and articulated hand shape recognition on a fine-grained hand shape taxonomy featuring over 60 different hand shapes. We clearly outperform the state-of-the-art on all data sets and observe significantly faster convergence using the parallel alignment approach.

Subfields: Computer Vision and Pattern Recognition, Developmental and Educational Psychology, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/964502ea316fe9049c464a0925e3dc4246023a0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Developmental and Educational Psychology

In Human-Computer Interaction

Citations in the corpus

MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language (2018, methodology): “showed large improvements embedding 2D-CNNs in HMMs [38, 40], related works with 3D-CNNs exist [6, 32] and weakly supervised multistream modeling is presented in [41].”
Improving Sign Language Translation with Monolingual Data by Sign Back-Translation (2021, neutral): “Due to the signiﬁcant social impact and the cross-modality challenge, sign language understanding has been attracting more and more research attention [1, 4, 10, 17, 22, 26, 37, 48].”
Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation (2020, methodology): “We also use an Inception [64] network which was pretrained for learning sign language recognition in a CNN+LSTM+HMM setup [39].”