Multimodal Learning

Vision-language models, CLIP, text-to-image, cross-modal representation.

A topic in the Science corpus, drawn from 1,221 papers.

Trusted voices in Multimodal Learning

Ordered by the trust Multimodal Learning's own network concentrates on each researcher, most first.

  1. Li Fei-Fei — trusted by 1,083 researchers here
  2. Andrej Karpathy — trusted by 897 researchers here
  3. Devi Parikh — trusted by 792 researchers here
  4. Dhruv Batra — trusted by 695 researchers here
  5. Ryan Kiros — trusted by 584 researchers here
  6. Richard S. Zemel — trusted by 545 researchers here
  7. Jiasen Lu — trusted by 479 researchers here
  8. Ruslan Salakhutdinov — trusted by 470 researchers here
  9. Mohit Bansal — trusted by 437 researchers here
  10. Tamara L. Berg — trusted by 354 researchers here
  11. Marcus Rohrbach — trusted by 365 researchers here
  12. Hao Hao Tan — trusted by 370 researchers here

Most-cited papers in Multimodal Learning

Ordered by the recorded citation count from the source corpus; a citation is not an endorsement.

  1. Learning Transferable Visual Models From Natural Language Supervision (2021, International Conference on Machine Learning) — 52,559 citations
  2. The Llama 3 Herd of Models (2024) — 17,085 citations
  3. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015) — 10,867 citations
  4. Visual Instruction Tuning (2023, Neural Information Processing Systems) — 10,396 citations
  5. Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, arXiv.org) — 9,210 citations
  6. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023) — 8,665 citations
  7. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022) — 8,628 citations
  8. Adding Conditional Control to Text-to-Image Diffusion Models (2023) — 7,354 citations
  9. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022) — 7,032 citations
  10. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016) — 6,638 citations
  11. Show and tell: A neural image caption generator (2014) — 6,597 citations
  12. Zero-Shot Text-to-Image Generation (2021) — 6,590 citations
  13. VQA: Visual Question Answering (2015) — 6,567 citations
  14. Flamingo: a Visual Language Model for Few-Shot Learning (2022, Neural Information Processing Systems) — 6,220 citations
  15. Deep Visual-Semantic Alignments for Generating Image Descriptions (2014) — 6,086 citations
  16. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021) — 5,764 citations
  17. Improved Baselines with Visual Instruction Tuning (2023) — 5,499 citations
  18. LAION-5B: An open large-scale dataset for training next generation image-text models (2022) — 5,389 citations
  19. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019) — 4,585 citations
  20. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023) — 4,547 citations
  21. Multimodal Machine Learning: A Survey and Taxonomy (2017) — 4,410 citations
  22. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022) — 4,398 citations
  23. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016) — 4,360 citations
  24. FiLM: Visual Reasoning with a General Conditioning Layer (2017) — 4,130 citations
  25. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (2024, arXiv.org) — 3,818 citations
  26. DreamFusion: Text-to-3D using 2D Diffusion (2022) — 3,724 citations
  27. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023) — 3,705 citations
  28. Multimodal Deep Learning (2011) — 3,550 citations
  29. Sigmoid Loss for Language Image Pre-Training (2023) — 3,447 citations
  30. Generative Adversarial Text to Image Synthesis (2016) — 3,433 citations
  31. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (2023) — 3,250 citations
  32. DeViSE: A Deep Visual-Semantic Embedding Model (2013) — 3,053 citations
  33. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018) — 3,041 citations
  34. CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021) — 3,009 citations
  35. LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019) — 2,971 citations
  36. PaLM-E: An Embodied Multimodal Language Model (2023) — 2,915 citations
  37. Prompt-to-Prompt Image Editing with Cross Attention Control (2022) — 2,848 citations
  38. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021) — 2,847 citations
  39. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022) — 2,832 citations
  40. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015) — 2,762 citations
  41. UNITER: UNiversal Image-TExt Representation Learning (2019) — 2,659 citations
  42. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015) — 2,641 citations
  43. MSR-VTT: A Large Video Description Dataset for Bridging Video and Language (2016) — 2,526 citations
  44. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (2022) — 2,503 citations
  45. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023) — 2,391 citations
  46. VisualBERT: A Simple and Performant Baseline for Vision and Language (2019) — 2,355 citations
  47. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021) — 2,347 citations
  48. SPICE: Semantic Propositional Image Caption Evaluation (2016) — 2,344 citations
  49. Conditional Prompt Learning for Vision-Language Models (2022) — 2,313 citations
  50. MMBench: Is Your Multi-modal Model an All-around Player? (2023) — 2,293 citations

Most active researchers

By papers in this topic: a volume measure, not trust.