KAT: A Knowledge Augmented Transformer for Vision-and-Language

By Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander G. Hauptmann, Yonatan Bisk, Jianfeng Gao. 2021

220 citations recorded by the source corpus.

The primary focus of recent work with large-scale transformers has been on optimizing the amount of information packed into the model’s parameters. In this work, we ask a complementary question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6% absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. Additionally, explicit knowledge integration improves interpretability of model predictions in our analysis.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[34] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao.”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, neutral): “TRiG [14] 50.5 KAT [17] 54.4 RA-VQA [33] 54.5 REVIVE [34] 58.0 PromptCap [22] 58.8 Z S PNP-VQA [54] 35.9 PICa [63] 43.3 BLIP-2 [31] 45.9 Flamingo [1] 50.6 ViperGPT (ours) 51.9”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “On OKVQA, it is likely that the benefit of having ViT-e cannot be exploited by the mT5-Large enc-dec as much as that by the mT5-XXL on VQA tasks, which require stronger language-understanding capabilities than Image Captioning tasks.”
REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering (2022, methodology): “Inspired by KAT [5], we consider both the explicit knowledge and implicit knowledge.”
Retrieval Augmented Visual Question Answering with Outside Knowledge (2022, methodology): “KB-VQA can access both structured data, such as ConceptNet and other KGs (Narasimhan et al., 2018a; Garderes et al., 2020; Li et al., 2020a; Wu et al., 2022; Marino et al., 2021), as well as unstructured data such as Wikipedia passages (Wu et al., 2022; Gao et al., 2022; Gui et al., 2021).”
Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory (2022, methodology): “Retrieval-augmented models have attracted a fair amount of attention in some recent NLP [14, 17] and computer vision [13, 24] works.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “Many models have been proposed for tasks with text and/or image input and text output (Cho et al., 2021; Wang et al., 2022d; Li et al., 2022b; Wang et al., 2021; Kaiser et al., 2017; Sun et al., 2022; Chen et al., 2022d; Wang et al., 2022c).”
Rethinking with Retrieval: Faithful Large Language Model Inference (2022, methodology): “, 2017) for visual question answering (Gui et al., 2021).”
Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering (2022, background): “In addition, a concurrent work [13] takes advantages of GPT-3 to retrieve implicit knowledge.”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “Recent methods [26,36,41,64] found that LLMs like GPT-3 could serve as a knowledge base and use the LLM to answer the question directly.”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, background): “5 - KAT [18] Caption + Tags + Feature - - - 54.”