Learning to Generate Scene Graph from Natural Language Supervision

By Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, Yin Li. 2021

94 citations recorded by the source corpus.

Learning from image-text data has demonstrated recent success for many recognition tasks, yet is currently limited to visual features or individual visual concepts such as objects. In this paper, we propose one of the first methods that learn from image-sentence pairs to extract a graphical representation of localized objects and their relationships within an image, known as scene graph. To bridge the gap between images and texts, we leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph. Further, we design a Transformer-based model to predict these "pseudo" labels via a masked token prediction task. Learning from only image-sentence pairs, our model achieves 30% relative gain over a latest method trained with human-annotated unlocalized scene graphs. Our model also shows strong results for weakly and fully supervised scene graph generation. In addition, we explore an open-vocabulary setting for detecting scene graphs, and present the first result for open-set scene graph generation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Jianwei Yang
3. Lu Yuan
4. Pengchuan Zhang
5. Xiyang Dai
6. Zhe Gan
7. Anelia Angelova
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Luc Van Gool
3. Jianwei Yang
4. Dacheng Tao
5. Lu Yuan
6. Trevor Darrell
7. Ming-Hsuan Yang
8. Xiaogang Wang

In Multimodal Learning

1. Jianfeng Gao
2. Jianwei Yang
3. Lu Yuan
4. Pengchuan Zhang
5. Xiyang Dai
6. Anelia Angelova
7. A. Piergiovanni
8. Weicheng Kuo

In Natural Language Generation

1. Jianfeng Gao

Citations in the corpus

• RegionCLIP: Region-based Language-Image Pretraining (2021, background): “The most relevant work is learning visual representations from natural language, such as image tags [3, 8, 12, 25, 28] and text descriptions [11,23,43,53,62].”
• Multimodal Learning With Transformers: A Survey (2022, background): “, image & text to scene graph [242], Video Dialogue (text & audio & visual to text) [243], Mono Audio &”
• F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, background): “…the promise of representation learning from natural language associated with images, such as image tags (Chen & Gupta, 2015; Divvala et al., 2014; Joulin et al., 2016) or text descriptions (Desai & Johnson, 2021; He & Peng, 2017; Sariyildiz et al., 2020; Wang et al., 2009; Zhong et al., 2021).”