Task-Oriented Multi-User Semantic Communications

By Huiqiang Xie, Zhijin Qin, Xiaoming Tao, Khaled Ben Letaief. IEEE Journal on Selected Areas in Communications, 2021

440 citations recorded by the source corpus.

While semantic communications have shown the potential in the case of single-modal single-users, its applications to the multi-user scenario remain limited. In this paper, we investigate deep learning (DL) based multi-user semantic communication systems for transmitting single-modal data and multimodal data, respectively. We adopt three intelligent tasks, including, image retrieval, machine translation, and visual question answering (VQA) as the transmission goal of semantic communication systems. We propose a Transformer based framework to unify the structure of transmitters for different tasks. For the single-modal multi-user system, we propose two Transformer based models, named, DeepSC-IR and DeepSC-MT, to perform image retrieval and machine translation, respectively. In this case, DeepSC-IR is trained to optimize the distance in embedding space between images and DeepSC-MT is trained to minimize the semantic errors by recovering the semantic meaning of sentences. For the multimodal multi-user system, we develop a Transformer enabled model, named, DeepSC-VQA, for the VQA task by extracting text-image information at the transmitters and fusing it at the receiver. In particular, a novel layer-wise Transformer is designed to help fuse multimodal data by adding connection between each of the encoder and decoder layers. Numerical results show that the proposed models are superior to traditional communications in terms of the robustness to channels, computational complexity, transmission delay, and the task-execution performance at various task-specific metrics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7b55fcc871aff4c6cbb25597087e164349e8b504

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges (2022, positive): “Furthermore, in [87], the authors unify the semantic encoding structure for both image transmitter and text transmitter based on Transformer.”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, neutral): “This scheme is recently extended in [233].”
Multi-modal Deep Learning (2024, background): “Xie et al. [14] investigate deep learning-based multi-user semantic communication systems for transmitting single-modal and multimodal data.”
Semantic Communications: Principles and Challenges (2021, methodology): “As an extension of DeepSC, A Transformer based framework [72] has”