LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

By Zhao Yang, Yansong Tang, Kai Chen, Hengshuang Zhao, Philip H. S. Torr. 2021

526 citations recorded by the source corpus.

Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image. One of the key challenges behind this task is leveraging the referring expression for highlighting relevant positions in the image. A paradigm for tackling this problem is to leverage a powerful vision-language (“cross-madal”) decoder to fuse features independently extracted from a vision encoder and a language encoder. Recent methods have made remarkable advancements in this paradigm by exploiting Transformers as cross-modal decoders, concurrent to the Transformer's overwhelming success in many other vision-language tasks. Adopting a different approach in this work, we show that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network. By conducting cross-modal feature fusion in the visual feature encoding stage, we can leverage the well-proven correlation modeling power of a Transformer encoder for excavating helpful multi-modal context. This way, accurate segmentation results are readily harvested with a light-weight mask predictor. Without bells and whistles, our method surpasses the previous state-of-the-art methods on Ref CoCo, RefCOCO+, and G-Ref by large margins.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jianfeng Gao
3. Dacheng Tao
4. Chen Change Loy
5. Xiaogang Wang
6. Ming-Hsuan Yang
7. Trevor Darrell
8. Cordelia Schmid

In Multimodal Learning

1. Zhe Gan
2. Jianfeng Gao
3. Lijuan Wang
4. Chunyuan Li
5. Linjie Li
6. Jianfeng Wang
7. Jianwei Yang
8. Yong Jae Lee

Citations in the corpus

• Generalized Decoding for Pixel, Image, and Language (2022, methodology): “Since the emergence of vision transformers, works like LAVT [86] enhance the cross-modal interactions from the very beginning, which led to SoTA on RefCOCO [92], RefCOCO+ [92] and G-Ref [56,60].”
• Transformer-Based Visual Segmentation: A Survey (2023, methodology): “• Conditional Query Fusion From Language Feature: Several works [48], [48], [57], [195], [196], [197], [199], [200], [201], [222], [245], [246], [247] adopt conditional query fusion according to input language feature for both referring image segmentation (RIS) [48] and referring video object…”
• Unleashing Text-to-Image Diffusion Models for Visual Perception (2023, methodology): “We also find that when trained for only 1 epoch, our VPD also achieves better overall IoU than previous state-of-the-art LAVT [56].”
• GRES: Generalized Referring Expression Segmentation (2023, methodology): “It is known that cIoU favors larger objects [40,44].”
• LViT: Language Meets Vision Transformer in Medical Image Segmentation (2022, methodology): “Different from VLT, Language-Aware Vision Transformer (LAVT) [27] framework adopted an early fusion scheme for integrating linguistic features into visual features via a pixel-word attention mechanism, which can effectively exploit the Transformer encoder for modeling multi-modal context.”