AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

By Feng Wen, Zixuan Zhang, Tianyiyi He, Chengkuo Lee. Nature Communications, 2021

437 citations recorded by the source corpus.

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers. Though wearable gloves are widely used for gesture associated applications (e.g. sign language recognition), sentence identification of sign language remains a challenge. Here, the authors report AI-enabled recognition system helps barrier-free communication between signers and non-signers.

Subfields: Biomedical Engineering, Cognitive Neuroscience, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/8ee7ba1cc742cb5bfdeeabebcaa913e57a8ec42c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

  1. Bin Feng
  2. Yu Xiao
  3. Wenxian Wang
  4. Chengjie Du
  5. Tianming Sun
  6. Jin Peng
  7. Lei Liu
  8. Zehua Li

Citations in the corpus