Language-Conditioned Graph Networks for Relational Reasoning

By Ronghang Hu, Anna Rohrbach, Trevor Darrell, Kate Saenko. 2019

186 citations recorded by the source corpus.

Solving grounded language tasks often requires reasoning about relationships between objects in the context of a given task. For example, to answer the question "What color is the mug on the plate?" we must check the color of the specific mug that satisfies the "on" relationship with respect to the plate. Recent work has proposed various methods capable of complex relational reasoning. However, most of their power is in the inference structure, while the scene is represented with simple local appearance features. In this paper, we take an alternate approach and build contextualized representations for objects in a visual scene to support relational reasoning. We propose a general framework of Language-Conditioned Graph Networks (LCGN), where each node represents an object, and is described by a context-aware representation from related objects through iterative message passing conditioned on the textual input. E.g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction. We experimentally show that our LCGN approach effectively supports relational reasoning and improves performance across several tasks and datasets. Our code is available at http://ronghanghu.com/lcgn.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Graph Learning

Citations in the corpus

STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, methodology): “L-GCN [13] iteratively uses location-aware GCN to model object’s spatial-temporal relations.”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “LGCN [20] 55.8 LXMERT [51] 60.0 NSM [24] 63.0 CRF [39] 72. fully Pythonic implementation is compatibility with a wide range of existing tools, such as PyTorch JIT [42].”
Question Directed Graph Attention Network for Numerical Reasoning over Text (2020, background): “Some recent works (LCGN (Hu et al., 2019b), NMNs (Gupta et al., 2020), NumNet (Ran et al., 2019)) attaching reasoning capabilities to models shows a promising direction.”
When Radiology Report Generation Meets Knowledge Graph (2020, methodology): “As many existing works (Yao et al. 2018; Liang et al. 2018; Chen et al. 2018; Norcliffe-Brown, Vafeias, and Parisot 2018; Hu et al. 2019), we used the graph convolution as a means of message passing and node interaction.”
Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers (2019, background): “, 2017), and symbolic approaches for visual question answering (Mao et al., 2019; Hu et al., 2019), etc.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “, 2019) ResNet-152 LSTM FinalDestGraph ADAM 7 (Hu et al., 2019) ResNet-101 LSTM LCGN ADAM 7 (Mao et al.”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, background): “It is useful in a broad range of tasks, such as language-conditioned manipulation [9, 10, 11, 12, 13, 14, 15], visual language navigation [16, 17], and visual question answering [18, 19, 20, 21, 22, 23, 24].”
Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering (2020, background): “[ Wang et al. , 2019b; Hu et al. , 2019a] propose heterogeneous graph attention networks with dual-level attention mechanism.”
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text (2020, methodology): “Similar to the previous GNNs [45, 36, 32, 21, 47] for grounding related tasks, we utilize the GNN to obtain better features.”
Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader (2019, unknown): “with symbolic approaches for visual QA (Gan et al., 2017; Mao et al., 2019; Hu et al., 2019).”
Differentiable Scene Graphs (2019, background): “Extracting SGs from images provides a semantic representation that can later be used for reasoning, question answering [54, 19, 32], and image retrieval [22, 42].”
Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals (2021, unknown): “Besides, there are quite a few studies that adopt Graph Neural Network (GNN) [16, 33] to conduct explicit reasoning on graph structure [12, 31].”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, methodology): “To build contextualized representations for objects in a visual scene, recent studies have introduced graph networks to iteratively pass messages among linked object nodes [9, 18].”
Visual Grounding in Video for Unsupervised Word Translation (2020, methodology): “2 [27] Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko.”
Multimodal Graph Networks for Compositional Generalization in Visual Question Answering (2020, background): “[25] also propose a graph-based representation over objects in an image and use message passing algorithms to update object representations conditioned on text (e.”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, background): “The graph-based reasoning has great applications in computer vision tasks such as action recognition [32, 36], scene graph generation [42], referring expression grounding [23, 35], and visual question answering [16, 24] etc.”
Interpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision (2020, methodology): “To better model the interactions between multiple objects, labelled relation or question-conditioned graph representations for images are adopted, then a GCN [34] is used to implicitly infer the interactive representation of objects [43,52,54,22,36].”
Towards Lightweight Transformer Via Group-Wise Transformation for Vision-and-Language Tasks (2022, methodology): “For CLEVR, we follow the previous works [42], [82]–[85] to use grid features extracted by ResNet101 [22].”