Neural Sign Language Translation

By Necati Cihan Camgöz, Simon Hadfield, Oscar Koller, Hermann Ney, Richard Bowden. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

863 citations recorded by the source corpus.

Sign Language Recognition (SLR) has been an active research field for the last two decades. However, most research to date has considered SLR as a naive gesture recognition problem. SLR seeks to recognize a sequence of continuous signs but neglects the underlying rich grammatical and linguistic structures of sign language that differ from spoken language. In contrast, we introduce the Sign Language Translation (SLT) problem. Here, the objective is to generate spoken language translations from sign language videos, taking into account the different word orders and grammar. We formalize SLT in the framework of Neural Machine Translation (NMT) for both end-to-end and pretrained settings (using expert knowledge). This allows us to jointly learn the spatial representations, the underlying language model, and the mapping between sign and spoken language. To evaluate the performance of Neural SLT, we collected the first publicly available Continuous SLT dataset, RWTH-PHOENIX-Weather 2014T1. It provides spoken language translations and gloss level annotations for German Sign Language videos of weather broadcasts. Our dataset contains over .95M frames with >67K signs from a sign vocabulary of >1K and >99K words from a German vocabulary of >2.8K. We report quantitative and qualitative results for various SLT setups to underpin future research in this newly established field. The upper bound for translation performance is calculated at 19.26 BLEU-4, while our end-to-end frame-level and gloss-level tokenization networks were able to achieve 9.58 and 18.13 respectively.

Subfields: Computer Vision and Pattern Recognition, Developmental and Educational Psychology, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/644602c65a5d8f30e62be027eb7b47f7c335191a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Developmental and Educational Psychology

In Human-Computer Interaction

Citations in the corpus

How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language (2020, methodology): “RWTH-Phoenix-2014T [12] DGS 3k 11 9 7 X X 7 7 7”
Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos (2020, methodology): “We test on the real-life continuous sign language recognition corpus RWTH-PHOENIX-Weather 2014 T [52], the continuous sign language mouth shape data”
Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation (2020, methodology): “The goal of sign language translation is to either convert written language into a video of sign (production) [59, 60] or to extract an equivalent spoken language sentence from a video of someone performing continuous sign [9].”
MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language (2018, background): “• The RWTH-PHOENIX-Weather 2014 [26, 33] and RWTH-PHOENIX-Weather 2014 T [5] are large scale real-life sign language corpora that feature professional interpreters recorded from broadcast news.”
Improving Sign Language Translation with Monolingual Data by Sign Back-Translation (2021, neutral): “Due to the signiﬁcant social impact and the cross-modality challenge, sign language understanding has been attracting more and more research attention [1, 4, 10, 17, 22, 26, 37, 48].”
Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective (2019, methodology): “on method impacts both content and signer identity. For example, some corpora are formed of professional interpreters paid to interpret spoken content, such as news channels that provide interpreting [43,74,21]. Others are formed of expert signers paid to sign desired corpus content (e.g., [65,124,1”
Learning Individual Styles of Conversational Gesture (2019, methodology): “This includes methods that use video transcripts as a weak source of supervision [3], as well as recent methods based on CNNs [33, 24] and RNNs [13].”