Differentiable Scene Graphs
By Moshiko Raboh, Roei Herzig, Gal Chechik, Jonathan Berant, Amir Globerson. 2019
34 citations recorded by the source corpus.
Reasoning about complex visual scenes involves perception of entities and their relations. Scene Graphs (SGs) provide a natural representation for reasoning tasks, by assigning labels to both entities (nodes) and relations (edges). Reasoning systems based on SGs are typically trained in a two-step procedure: first, a model is trained to predict SGs from images, and next a separate model is trained to reason based on the predicted SGs. However, it would seem preferable to train such systems in an end-to-end manner. The challenge, which we address here is that scene-graph representations are non-differentiable and therefore it isn’t clear how to use them as intermediate components. Here we propose Differentiable Scene Graphs (DSGs), an image representation that is amenable to differentiable end-to-end optimization, and requires supervision only from the downstream tasks. DSGs provide a dense representation for all regions and pairs of regions, and do not spend modelling capacity on regions of the image that do not contain objects or relations of interest. We evaluate our model on the challenging task of identifying referring relationships (RR) in three benchmark datasets: Visual Genome, VRD and CLEVR. Using DSGs as an intermediate representation leads to new state-of-the-art performance. The full code is available at https://github.com/shikorab/DSG.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Hadar Serviansky
- Nimrod Segol
- Haggai Maron
- Yaron Lipman
- Jonathan Shlomi
- Xiaojun Chang
- Kyle Cranmer
- Pengzhen Ren
Citations in the corpus
- Set2Graph: Learning Graphs From Sets (2020, background): “Another direction is incorporating our network architecture in set models, with the goal of improving performance of general set tasks by constructing an intermediate latent graph representation, for example for constructing scene graphs as in [29].”
- A Comprehensive Survey of Scene Graphs: Generation and Application (2021, methodology): “Therefore, DSG (Differentiable Scene-Graphs) [113] are employed to solve the above obstacles.”