Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation

By Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, Rongrong Ji. Computer Vision and Pattern Recognition, 2020

370 citations recorded by the source corpus.

Referring expression comprehension (REC) and segmentation (RES) are two highly-related tasks, which both aim at identifying the referent according to a natural language expression. In this paper, we propose a novel Multi-task Collaborative Network (MCN) to achieve a joint learning of REC and RES for the first time. In MCN, RES can help REC to achieve better language-vision alignment, while REC can help RES to better locate the referent. In addition, we address a key challenge in this multi-task setup, i.e., the prediction conflict, with two innovative designs namely, Consistency Energy Maximization (CEM) and Adaptive Soft Non-Located Suppression (ASNLS). Specifically, CEM enables REC and RES to focus on similar visual regions by maximizing the consistency energy between two tasks. ASNLS supresses the response of unrelated regions in RES based on the prediction of REC. To validate our model, we conduct extensive experiments on three benchmark datasets of REC and RES, i.e., RefCOCO, RefCOCO+ and RefCOCOg. The experimental results report the significant performance gains of MCN over all existing methods, i.e., up to +7.13% for REC and +11.50% for RES over SOTA, which well confirm the validity of our model for joint REC and RES learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8d1bc83cc65d30e4619c49f53115012a209fd8c9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Amalgamating Knowledge from Heterogeneous Graph Neural Networks (2021, background): “A more recent work [21] further proposes a multi-task collaborative network that achieves joint learning of re-ferring expression comprehension and segmentation.”
Uncertainty-aware Joint Salient Object and Camouflaged Object Detection (2021, background): “[32] presented a multitask framework for referring expression comprehension and segmentation.”
Referring Expression Comprehension via Cross-Level Multi-Modal Fusion (2022, methodology): “Recent advancements in REC can be divided into two research directions: two-stage methods [3, 17, 18, 29, 54] and one-stage methods [8, 25, 27, 35, 51, 52].”
Improving One-stage Visual Grounding by Recursive Sub-query Construction (2020, background): “Recently, several works [47,5,37,20,28] propose a diﬀerent paradigm of one-stage visual grounding.”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “Similarly, other works leverage visual grounded-reasoning [39, 79] data in training, such as Shikra [6] and KOSMOS-2 [46].”
FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “We note that it is unclear how to use existing grounding models out-of-the-box for task unification due to the task-specific architectures, losses, and training strategies [13,21,51,55,88,89,91].”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “The encoded representations of different layers in the transformer are extracted and merged with a decoder via skip connections to get the ﬁnal 3D mask outputs.”
Vision-Language Transformer and Query Generation for Referring Segmentation (2021, methodology): “For example, [19] gives each word a weight and [29, 12] defines a set of labels, e.”
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, methodology): “The other one is single-stage modeling, [65, 76, 41, 64], which considers REC as a language-guided detection task.”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…77, 36, 19, 38, 51, 50, 20, 30], generic image segmentation [9, 8, 26, 29, 78, 88, 79, 25, 79, 16, 28], referring object detection and segmentation [41, 37, 86], object tracking [67, 84], image generation [75, 54, 48, 45, 23, 14, 52, 22, 82, 44], image editing [42, 1, 2, 53, 21], human-centric…”
CRIS: CLIP-Driven Referring Image Segmentation (2021, methodology): “Besides, MCN [30] designed a framework achieving impressive results.”
GRES: Generalized Referring Expression Segmentation (2023, methodology): “In the baseline model, RLA is replaced by point-wise multiplying region features and globally averaged language features, to achieve a basic feature fusion like previous works [5, 32].”