Entity-Enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding

By Xuejing Liu, Liang Li, Shuhui Wang, Zhengjun Zha, Dechao Meng, Qingming Huang. 2022

76 citations recorded by the source corpus.

Weakly supervised Referring Expression Grounding (REG) aims to ground a particular target in an image described by a language expression while lacking the correspondence between target and expression. Two main problems exist in weakly supervised REG. First, the lack of region-level annotations introduces ambiguities between proposals and queries. Second, most previous weakly supervised REG methods ignore the discriminative location and context of the referent, causing difficulties in distinguishing the target from other same-category objects. To address the above challenges, we design an entity-enhanced adaptive reconstruction network (EARN). Specifically, EARN includes three modules: entity enhancement, adaptive grounding, and collaborative reconstruction. In entity enhancement, we calculate semantic similarity as supervision to select the candidate proposals. Adaptive grounding calculates the ranking score of candidate proposals upon subject, location and context with hierarchical attention. Collaborative reconstruction measures the ranking result from three perspectives: adaptive reconstruction, language reconstruction and attribute classification. The adaptive mechanism helps to alleviate the variance of different referring expressions. Experiments on five datasets show EARN outperforms existing state-of-the-art methods. Qualitative results demonstrate that the proposed EARN can better handle the situation where multiple objects of a particular category are situated together.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022, background): “…is a long line of work in weakly supervised520 ReC, where at training time, pairs of referring ex-521 pressions and images are available but the ground-522 truth bounding boxes for each expression are not523 (Rohrbach et al., 2016; Liu et al., 2019; Zhang524 et al., 2018, 2020; Sun et al., 2021).”