Learning to Compose Dynamic Tree Structures for Visual Contexts

By Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, Wei David Liu. Computer Vision and Pattern Recognition, 2018

612 citations recorded by the source corpus.

We propose to compose dynamic tree structures that place the objects in an image into a visual context, helping visual reasoning tasks such as scene graph generation and visual Q&A. Our visual context tree model, dubbed VCTree, has two key advantages over existing structured object representations including chains and fully-connected graphs: 1) The efficient and expressive binary tree encodes the inherent parallel/hierarchical relationships among objects, e.g., ``clothes'' and ``pants'' are usually co-occur and belong to ``person''; 2) the dynamic structure varies from image to image and task to task, allowing more content-/task-specific message passing among objects. To construct a VCTree, we design a score function that calculates the task-dependent validity between each object pair, and the tree is the binary version of the maximum spanning tree from the score matrix. Then, visual contexts are encoded by bidirectional TreeLSTM and decoded by task-specific models. We develop a hybrid learning procedure which integrates end-task supervised learning and the tree structure reinforcement learning, where the former's evaluation result serves as a self-critic for the latter's structure exploration. Experimental results on two benchmarks, which require reasoning over contexts: Visual Genome for scene graph generation and VQA2.0 for visual Q&A, show that VCTree outperforms state-of-the-art results while discovering interpretable visual context structures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/88ec56eee6a787ccda0cf7fbfbec48c6b4ad68fe

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, neutral): “Visual context is crucial for recognition [11, 45, 53].”
Learning to Collocate Neural Modules for Image Captioning (2019, background): “Unlike the widely-used neural module networks in visual Q&A, where the language (i.e., question) is fully observable, CNM for captioning is more challenging as the language is being generated and thus is partially observable.”
Recursive Visual Attention in Visual Dialog (2018, background): “Different from one-round VQA, visual dialog is a multi-round conversation about an image.”
Spatial-Temporal Transformer for Dynamic Scene Graph Generation (2021, background): “The applications include image retrieval [46], image captioning [1, 45], VQA [51, 25] and image generation [24, 19].”
Image Scene Graph Generation (SGG) Benchmark (2021, neutral): “0) No No SGG-Pytorch [15] VG IMP,NM,VCTree [16] 12/13/2020 (1.”
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models (2021, background): “In the literature, since the ground-truth relations cannot be exhaustively annotated during evaluation, to avoid false negatives, previous works typically score the triplets and evaluate the recall of top-N triplets (Xu et al., 2017; Zellers et al., 2018; Chen et al., 2019; Tang et al., 2019).”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “sets, [85] proposed a new unbiased metric (Mean Recall@K) to average the recall of all types of relations instead of all samples, and avoiding the neglect of relations with only few”
Unbiased Scene Graph Generation From Biased Training (2020, positive): “5% lower than the state-of-the-art [5, 53, 72].”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “[72] uses Bi-TreeLSTMs (bidirectional TreeLSTMs) [84] to encode visual context in feature extraction (Link I ! X).”
Interpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision (2020, methodology): “To better model the interactions between multiple objects, labelled relation or question-conditioned graph representations for images are adopted, then a GCN [34] is used to implicitly infer the interactive representation of objects [43,52,54,22,36].”
Entangled Transformer for Image Captioning (2019, background): “Tang et al. [36] leverage scene graph to align the relations between vision and language.”
Counterfactual Samples Synthesizing for Robust Visual Question Answering (2020, methodology): “Then both V and Q are fed into the model f vqa to predict answer distributions: Model f vqa typically contains an attention mechanism [13, 29, 41], and it is trained with cross-entropy loss [38, 11].”
GPS-Net: Graph Property Sensing Network for Scene Graph Generation (2020, methodology): “Another advantage of GPS-Net over VCTREE and CMAT is that GPS-Net is much more efﬁcient, as the two meth-ods adopt policy gradient for optimization, which is time-consuming [46].”
Two Causal Principles for Improving Visual Dialog (2019, neutral): “, image captioning [46, 44, 4] and VQA [5, 38, 37, 36].”
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, neutral): “Scene graph generation (Li et al., 2017; Tang et al., 2018; Yang et al., 2022) analyzes the relationships be-tween regions by the graph.”
Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation (2020, methodology): “To address issue (a), the predicate-normalized metric, mean recall (mR@K) [6, 31] and weighted mR@K were introduced [45].”
SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, methodology): “The width of Region [1] is 7.73 feet.”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “By observing the potential of exploiting scene graphs in vision tasks, a variety of approaches are proposed to improve the scene graph generation from images [57, 56, 52, 51, 49, 42, 6].”
Learning to Generate Scene Graph from Natural Language Supervision (2021, methodology): “SGDet captures both the localization and classification performance using metrics of Recall@K (R@K) [29, 52] and mean Recall@K (mR@K) [5, 46].”