SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition

By Necati Cihan Camgöz, Simon Hadfield, Oscar Koller, Richard Bowden. IEEE International Conference on Computer Vision, 2017

335 citations recorded by the source corpus.

We propose a novel deep learning approach to solve simultaneous alignment and recognition problems (referred to as “Sequence-to-sequence” learning). We decompose the problem into a series of specialised expert systems referred to as SubUNets. The spatio-temporal relationships between these SubUNets are then modelled to solve the task, while remaining trainable end-to-end. The approach mimics human learning and educational techniques, and has a number of significant advantages. SubUNets allow us to inject domain-specific expert knowledge into the system regarding suitable intermediate representations. They also allow us to implicitly perform transfer learning between different interrelated tasks, which also allows us to exploit a wider range of more varied data sources. In our experiments we demonstrate that each of these properties serves to significantly improve the performance of the overarching recognition system, by better constraining the learning problem. The proposed techniques are demonstrated in the challenging domain of sign language recognition. We demonstrate state-of-the-art performance on hand-shape recognition (outperforming previous techniques by more than 30%). Furthermore, we are able to obtain comparable sign recognition rates to previous research, without the need for an alignment step to segment out the signs for recognition.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/a10cd29fec9dca66250dbde19db5e831f7ce6406

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation (2020, methodology): “Such methods can be grouped under the banner of CSLR [40, 8].”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, background): “The field of 3D hand pose estimation has advanced rapidly, both in terms of accuracy [2, 4, 5, 6, 7, 26, 28, 32, 38, 45, 48, 49, 53, 54, 57, 60, 62] and dataset quality [10, 22, 43, 47, 52, 59].”
Improving Sign Language Translation with Monolingual Data by Sign Back-Translation (2021, methodology): “Method Dev Test del/ins WER del/ins WER SubUNets [9] 14.”
Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos (2020, neutral): “CNN-LSTM with CTC [74] yes no 40.”