Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?

By Tobias Norlund, Lovisa Hagström, Richard Johansson. 2021

26 citations recorded by the source corpus.

Large language models are known to suffer from the hallucination problem in that they are prone to output statements that are false or inconsistent, indicating a lack of knowledge. A proposed solution to this is to provide the model with additional data modalities that complements the knowledge obtained through text. We investigate the use of visual data to complement the knowledge of large language models by proposing a method for evaluating visual knowledge transfer to text for uni- or multimodal language models. The method is based on two steps, 1) a novel task querying for knowledge of memory colors, i.e. typical colors of well-known objects, and 2) filtering of model training data to clearly separate knowledge contributions. Additionally, we introduce a model architecture that involves a visual imagination step and evaluate it with our proposed method. We find that our method can successfully be used to measure visual knowledge transfer capabilities in models and that our novel model architecture shows promising results for leveraging multimodal knowledge in a unimodal setting.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Furu Wei
2. Jianfeng Gao
3. Li Dong
4. Wenhui Wang
5. Shaohan Huang
6. Shuming Ma
7. Luc Van Gool
8. Dacheng Tao

In Natural Language Processing

1. Furu Wei
2. Li Dong
3. Jianfeng Gao
4. Xiaodong Liu
5. Shaohan Huang
6. Wenhui Wang
7. Shuming Ma
8. Xia Song

In Multimodal Learning

1. Li Dong
2. Furu Wei
3. Jianfeng Gao
4. Wenhui Wang
5. Shaohan Huang
6. Shuming Ma
7. Saksham Singhal
8. Owais Khan Mohammed

Citations in the corpus

• Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accu”
• Visually-Augmented Language Modeling (2022, methodology): “Norlund et al. (2021) proposed this dataset for evaluating visual knowledge transfer in multi-modal language models.”