Multi-Modal Knowledge Graph Construction and Application: A Survey

By Xiangru Zhu, Zhixu Li, Xiaodan Wang, Xueyao Jiang, Penglei Sun, Xuwu Wang, Yanghua Xiao, Nicholas Jing Yuan. IEEE Transactions on Knowledge and Data Engineering, 2022

296 citations recorded by the source corpus.

Recent years have witnessed the resurgence of knowledge engineering which is featured by the fast growth of knowledge graphs. However, most of existing knowledge graphs are represented with pure symbols, which hurts the machine's capability to understand the real world. The multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. The results of this endeavor are Multi-modal Knowledge Graphs (MMKGs). In this survey on MMKGs constructed by texts and images, we first give definitions of MMKGs, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strengths and weaknesses of different solutions. We finalize this survey with open research problems relevant to MMKGs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fa350b1089db1f8ab97bb72287b37ed4748c89cf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Feng Xia
2. Linhao Luo
3. Ciyuan Peng
4. Xindong Wu
5. Francesco Osborne
6. P. Xu
7. Chen Chen
8. Mehdi Naseriparsa

In Computer Vision and Pattern Recognition

1. Xindong Wu
2. P. Xu
3. Feng Xia
4. Xiatian Zhu
5. Shirui Pan
6. David A. Clifton
7. Yufei Wang
8. Dacheng Tao

Citations in the corpus

• Multimodal Learning With Transformers: A Survey (2022, background): “language [27], sign language [28], [29], [30], point cloud [31], symbolic knowledge (graph) [32], [33], multimodal knowledge graph [34], sketch drawing [35], [36], [37], [38], 3D object/scene [39], [40], [41], document [42], [43], programming code [44] and Abstract Syntax Tree (AST) – a kind of grap”
• Knowledge Graphs: Opportunities and Challenges (2023, background): “Notably, multi-hop reasoning on massive knowledge graphs is one of the challenging tasks (Zhu et al. 2022).”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “Unlike conventional knowledge graphs that only contain textual information, multi-modal knowledge graphs represent facts in multiple modalities such as images, sounds, and videos [83].”
• A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal (2022, unknown): “Typically, there are two multi-modal KG [21], i.e., N-MMKG and A-MMKG (See Fig.”