CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

By Andreas Furst, Elisabeth Rumetshofer, Viet-Hung Tran, Hubert Ramsauer, Fei Tang, Johannes Lehner, David P. Kreil, Michael Kopp, Günter Klambauer, Angela Bitto-Nemling and 1 more. 2021

129 citations recorded by the source corpus.

CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3. CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on particular tasks. Though CLIP excels at zero-shot transfer learning, it suffers from an explaining away problem, that is, it focuses on one or few features, while neglecting other relevant features. This problem is caused by insufficiently extracting the covariance structure in the original multi-modal data. We suggest to use modern Hopfield networks to tackle the problem of explaining away. Their retrieved embeddings have an enriched covariance structure derived from co-occurrences of features in the stored embeddings. However, modern Hopfield networks increase the saturation effect of the InfoNCE objective which hampers learning. We propose to use the InfoLOOB objective to mitigate this saturation effect. We introduce the novel"Contrastive Leave One Out Boost"(CLOOB), which uses modern Hopfield networks for covariance enrichment together with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after pre-training on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Prafulla Dhariwal
2. Chen Change Loy
3. Ziwei Liu
4. Jingkang Yang
5. Sepp Hochreiter
6. Kaiyang Zhou

In Computer Vision and Pattern Recognition

1. Chen Change Loy
2. Luc Van Gool
3. Ziwei Liu
4. Ming-Hsuan Yang
5. Dacheng Tao
6. Yu Qiao
7. Xiaogang Wang
8. Trevor Darrell

In Multimodal Learning

1. Ziwei Liu
2. Alex Nichol
3. Chen Change Loy
4. Kaiyang Zhou
5. Prafulla Dhariwal
6. Mark Chen
7. Jingkang Yang
8. Aditya Ramesh

In Representation & Self-Supervised Learning

1. Ziwei Liu
2. Jingkang Yang
3. Chen Change Loy
4. Kaiyang Zhou
5. Alex Nichol
6. Aditya Ramesh
7. Mark Chen
8. Casey Chu

Citations in the corpus

• Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
• Learning to Prompt for Vision-Language Models (2021, background): “diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
• Conditional Prompt Learning for Vision-Language Models (2022, methodology): “Recent research in large-scale vision-language pretraining has achieved striking performance in zero-shot image recognition [13,24,33,40], demonstrating a potential in learning open-world visual concepts for such a paradigm.”
• History Compression via Language Models in Reinforcement Learning (2022, background): “FrozenHopfield is an instance of a MHN (Ramsauer et al., 2021; Fürst et al., 2021; Schäfl et al., 2022) that can associate image observations (or any other input) with pretrained token embeddings (Mikolov et al.”