MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

By Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun and 12 more. 2023

2,391 citations recorded by the source corpus.

We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and text-books, covering six core disciplines: Art & Design, Busi-ness, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly het-erogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 28 open-source LMMs as well as the propri-etary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Quoc V. Le
  4. Graham Neubig
  5. Yoshua Bengio
  6. Percy Liang
  7. Yejin Choi
  8. Sergey Levine

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Xiaogang Wang
  3. Dacheng Tao
  4. Yu Qiao
  5. Trevor Darrell
  6. Ming-Hsuan Yang
  7. Jitendra Malik
  8. Alan L. Yuille

In Multimodal Learning

  1. Yu Qiao
  2. Peng Gao
  3. Renrui Zhang
  4. Saining Xie
  5. Pan Lu
  6. Penghao Wu
  7. Jingren Zhou
  8. Dahua Lin

In Evaluation & Benchmarks

  1. Graham Neubig
  2. Zhiyuan Liu
  3. Matt Fredrikson
  4. Maosong Sun
  5. Xu Han
  6. Noah A. Smith
  7. Bo Li
  8. Yu Qiao

Citations in the corpus