Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
By Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, Richard Bowden. Computer Vision and Pattern Recognition, 2020
771 citations recorded by the source corpus.
Prior work on Sign Language Translation has shown that having a mid-level sign gloss representation (effectively recognizing the individual signs) improves the translation performance drastically. In fact, the current state-of-the-art in translation requires gloss level tokenization in order to work. We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation while being trainable in an end-to-end manner. This is achieved by using a Connectionist Temporal Classification (CTC) loss to bind the recognition and translation problems into a single unified architecture. This joint approach does not require any ground-truth timing information, simultaneously solving two co-dependant sequence-to-sequence learning problems and leads to significant performance gains. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset. We report state-of-the-art sign language recognition and translation results achieved by our Sign Language Transformers. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models, in some cases more than doubling the performance (9.58 vs. 21.80 BLEU-4 Score). We also share new baseline translation results using transformer networks for several other text-to-text sign language translation tasks.
Subfields: Computer Vision and Pattern Recognition, Developmental and Educational Psychology, Human-Computer Interaction
View Paper: https://www.semanticscholar.org/paper/05dcdfece56d1869895f53ed581d8ad64118c05f
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Yichen Wei
- Tiancai Wang
- Luc Van Gool
- Ming-Hsuan Yang
- Xiaogang Wang
- Dacheng Tao
- Chen Change Loy
- Yi Yang
In Human-Computer Interaction
Citations in the corpus
- Multimodal Learning With Transformers: A Survey (2022, background): “, lip reading [84], sign language translation [25], [26], [85]), human-computer interaction [86], healthcare AI [87], [88], surveillance AI [89], etc.”
- MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, background): “Since that, it was gradually introduced to many fields, such as speech processing [13,7] and computer vision [34,5].”