Pixels to Graphs by Associative Embedding

By Alejandro Newell, Jia Deng. 2017

242 citations recorded by the source corpus.

Graphs are a useful abstraction of image content. Not only can graphs represent details about individual objects in a scene but they can capture the interactions between pairs of objects. We present a method for training a convolutional neural network such that it takes in an input image and produces a full graph definition. This is done end-to-end in a single stage with the use of associative embeddings. The network learns to simultaneously identify all of the elements that make up a graph and piece them together. We benchmark on the Visual Genome dataset, and demonstrate state-of-the-art performance on the challenging task of scene graph generation.

Subfields: Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

Object-Centric Image Generation from Layouts (2020, background): “2015) and image to scene graph prediction (Lu et al. 2016; Newell and Deng 2017).”
CornerNet: Detecting Objects as Paired Keypoints (2018, positive): “As in Newell and Deng (2017), we use the “pull” loss to train the network to group the corners and the “push” loss to separate the corners: Lpull = 1 N N∑ k=1 [ (etk − ek) 2 + (ebk − ek) 2 ] , (4) Lpush = 1 N(N − 1) N∑ k=1 N∑ j=1 j 6=k max (0, ∆− |ek − ej”
Image Generation from Scene Graphs (2018, methodology): “Scene graphs have been used for image retrieval [22] and to evaluate image captioning [1]; some work converts sentences to scene graphs [47] or predicts grounded scene graphs for images [32, 36, 57, 58].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Given its versatile utility, the task of scene graph prediction has resulted in a series of publications [14, 30, 43, 46, 48, 49, 59, 77, 78, 85] that have explored reinforcement learning [49], structured prediction [16, 40, 70], utilizing object attributes [20, 61], sequential prediction [59], few-”
Graph R-CNN for Scene Graph Generation (2018, methodology): “A number of approaches have been proposed for the detection of both objects and their relationships [2,17–19, 23, 26 , Most scene graph generation methods also include some mechanisms for context propagation and reasoning over a candidate scene graph in order to reﬁne the ﬁnal labeling.”
PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection (2019, background): “Different from the general visual relationship detection [19, 29, 20, 13, 32], the subject of the triplet is fixed as human while the interaction is action.”
Learning to Compose Dynamic Tree Structures for Visual Contexts (2018, background): “(2) AssocEmbed [34] assembles implicit contextual features by stacked hourglass backbone [35].”
LinkNet: Relational Embedding for Scene Graph (2018, unknown): “…using iterative message passing [33]; utilizing associative embedding to simultaneously identify nodes and edges of graph and piece them together [22]; extending the idea of the message passing from [33] with additional RPN in order to propose regions for captioning and solve tasks jointly [18];…”
Knowledge-Embedded Routing Network for Scene Graph Generation (2019, methodology): “Latterly, lots of attention [19, 30, 6, 16, 23, 33] was drawn to the visual relationship detection task under a more”
SOGNet: Scene Overlap Graph Network for Panoptic Segmentation (2019, methodology): “2017), associative embedding (Newell and Deng 2017), and introducing an efficient module (Santoro et al.”
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020, background): “Later, Visual Genome [19], a large scale dataset with scene graph annotations on images, gave rise to a line of deep learning based advances on scene graph prediction from images [48, 11, 37, 51, 25, 49, 24, 33].”
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering (2019, unknown): “Lately, generating scene graphs from images has achieved notable progress [39, 44, 55, 67, 69, 77].”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, methodology): “Following [22,23], we ignore the non-peak output locations rather than forcing them to be zero.”
PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph (2019, background): “, image retrieval [18], image captioning evaluation [19], sentence-scene graph translation [20], and image-based scene graph prediction [21, 22, 23].”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “ple, previous SGG work [22], [30], [31], [75], [98] often”