Sherlock: Scalable Fact Learning in Images

By Mohamed Elhoseiny, Scott D. Cohen, Walter Chang, Brian L. Price, Ahmed M. Elgammal. 2015

27 citations recorded by the source corpus.

We study scalable and uniform understanding of facts in images. Existing visual recognition systems are typically modeled differently for each fact type such as objects, actions, and interactions. We propose a setting where all these facts can be modeled simultaneously with a capacity to understand an unbounded number of facts in a structured way. The training data comes as structured facts in images, including (1) objects (e.g., ), (2) attributes (e.g., ), (3) actions (e.g., ), and (4) interactions (e.g., ). Each fact has a semantic language view (e.g., < boy, playing>) and a visual view (an image with this fact). We show that learning visual facts in a structured way enables not only a uniform but also generalizable visual understanding. We propose and investigate recent and strong approaches from the multiview learning literature and also introduce two learning representation models as potential baselines. We applied the investigated methods on several datasets that we augmented with structured facts and a large scale dataset of more than 202,000 facts and 814,000 images. Our experiments show the advantage of relating facts by the structure by the proposed models compared to the designed baselines on bidirectional fact retrieval.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Memory Aware Synapses: Learning what (not) to forget (2017, methodology): “The base model We build on the model introduced recently in [3]: a CNN model based on the VGG-16 architecture [27] pretrained on ImageNet.”
Weakly-Supervised Learning of Visual Relations (2017, background): “Finally, we also address the problem to evaluate accurately due to missing annotations also pointed out in [10, 31].”
Exploring the Challenges towards Lifelong Fact Learning (2018, methodology): “This can help semantically similar concepts to learn better from one another, as shown in [31,7] in non LLL scenarios.”
Continual Learning in Neural Networks (2019, methodology): “The last fully connected layer forks in three final layers enabling the model to have three separated and structured outputs for Subject, Predicate and Object as in [35].”
Memory Aware Synapses: Learning what (not) to forget (2018, methodology): “The last fully connected layer forks in three final layers enabling the model to have three separated and structured outputs for Subject, Predicate and Object as in [4].”
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation (2018, background): “In the early stage, most of the works targeted on using speciﬁc types of visual relations, such as spatial relations [5, 10, 14, 20, 26, 30] and actions ( i . e . interactions between objects) [1, 9, 11, 16, 19, 46, 47, 50, 56, 57, 60].”
Large-Scale Visual Relationship Understanding (2018, background): “For ﬁne-grained objects like birds and ﬂowers, several works adopted Wikipedia Articles to guide zero-shot recognition since they are easy-collect language and provides more signal for recognition (e.g., [8, 7, 1, 6].”
Detecting Visual Relationships with Deep Relational Networks (2017, methodology): “In most of these studies, relationships are usually extracted using simple heuristics or hand-crafted features, and used as an auxiliary components to facilitate other tasks, such as object recognition [25–33], image classiﬁcation and retrieval [34,35], scene understanding 1 code available at…”