Image-embodied Knowledge Representation Learning

By Ruobing Xie, Zhiyuan Liu, Huanbo Luan, Maosong Sun. International Joint Conference on Artificial Intelligence, 2016

303 citations recorded by the source corpus.

Entity images could provide significant visual information for knowledge representation learning. Most conventional methods learn knowledge representations merely from structured triples, ignoring rich visual information extracted from entity images. In this paper, we propose a novel Image-embodied Knowledge Representation Learning model (IKRL), where knowledge representations are learned with both triple facts and images. More specifically, we first construct representations for all images of an entity with a neural image encoder. These image representations are then integrated into an aggregated image-based representation via an attention-based method. We evaluate our IKRL models on knowledge graph completion and triple classification. Experimental results demonstrate that our models outperform all baselines on both tasks, which indicates the significance of visual information for knowledge representations and the capability of our models in learning knowledge representations with images.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/657703c9914ce785649c67374a0e8860a1b4321c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Philip S. Yu
2. Shirui Pan
3. Fuzheng Zhang
4. Erik Cambria
5. Zhongyuan Wang
6. Shaoxiong Ji
7. Yue Liu
8. Siwei Wang

In Computer Vision and Pattern Recognition

1. Shirui Pan
2. Philip S. Yu
3. Yue Liu
4. Siwei Wang
5. Zhixu Li
6. Wenxuan Tu
7. Penglei Sun
8. Kun Zhou

Citations in the corpus

• Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, methodology): “To prepare the training data for the classification model, correct triplets are corrupted by replacing one of the h, r, t to generate negative data [138], [143], [144].”
• Multi-modal Knowledge Graphs for Recommender Systems (2020, methodology): “These works have proved that the multi-modal knowledge graph plays an important role in knowledge graph completion and triple classification [5, 17, 30].”
• A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal (2022, methodology): “Inspired by it, IKRL [240] first designs an attention-based neural network to consider visual information in entity images.”
• A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, methodology): “For KRL with auxiliary information and other downstream knowledge-aware applications, texts and images are also collected, for example, WN18-IMG [63] with sampled images and textual relation extraction dataset including SemEval 2010 dataset, NYT [198] and GoogleRE10.”
• Knowledge Graph Embedding for Link Prediction (2020, methodology): “We refer the reader to works discussing how to leverage additional sources of information, such as textual captions [58],[65],[3], images [70] or pre-computed rules [20]; see [18] for a survey exclusive to these models.”