Cross-lingual and Multilingual CLIP

By Fredrik Carlsson, Philipp Eisen, Faton Rekathati, Magnus Sahlgren. 2022

74 citations recorded by the source corpus.

The long-standing endeavor of relating the textual and the visual domain recently underwent a pivotal breakthrough, as OpenAI released CLIP. This model distinguishes how well an English text corresponds with a given image with unprecedented accuracy. Trained via a contrastive learning objective over a huge dataset of 400M of images and captions, it is a work that is not easily replicated, especially for low resource languages. Capitalizing on the modularization of the CLIP architecture, we propose to use cross-lingual teacher learning to re-train the textual encoder for various non-English languages. Our method requires no image data and relies entirely on machine translation which removes the need for data in the target language. We find that our method can efficiently train a new textual encoder with relatively low computational cost, whilst still outperforming previous baselines on multilingual image-text retrieval.

Subfields: Artificial Intelligence, Developmental and Educational Psychology, Experimental and Cognitive Psychology, Multimodal Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Multimodal Learning

Citations in the corpus

LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “For languages other than English, we utilized the multi-lingual CLIP ViT-B/32 from Carlsson et al. [10].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “Additionally, as shown in Table 6b, our model showcases robust multilingual capabilities, outperforming competing models [14, 24, 51, 126] on the multilingual ImageNet-1K benchmark.”