GPT-4V(ision) System Card

2023

Citations in the corpus

• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, methodology): “In light of the increasing interest in multimodal LLMs (OpenAI, 2023c), we also fine-tune Llama-2-7b-Chat on LLaVA-Instruct (Liu et al., 2023a), integrating the language model with a CLIP visual encoder (Radford et al., 2021).”
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, unknown): “To verify the effectiveness of LLM-based evaluation for LMM predictions, we select the outputs from MMReAct-GPT-4 on 138 objective questions, which can be objectively annotated by humans.”
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, methodology): “We evaluate the models on M ATH V ISTA under three setups: (a) Text-Only LLMs including ChatGPT (OpenAI, 2022), GPT-4 (OpenAI, 2023a), and Claude-2 (Anthropic, 2023) in zero-shot and two-shot settings with Chain-of-Thought (CoT) (Wei et al., 2022b) and Program-of-Thought (PoT) (Chen et al., 2022b),…”
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, background): “In the past few years, Multi-modal Large Language Models (MLLMs) [1, 16, 26, 39, 42, 47, 59, 104] have gradually driven the advance in vision-language learning, by plugging visual encoders within various pretrained LLMs [10, 15, 58, Figure 1.”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, unknown): “Importantly, the videos vary significantly in length, ranging from 11 seconds to 1 hour, specifically evaluating the adaptability of MLLMs across varying temporal contexts.”
• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, methodology): “Besides textual attacks, visual jailbreak also threatens the safety of multimodal models (e.g., GPT-4V [101], and Sora [3]).”
• PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation (2024, methodology): “Additionally, in our AI preference study, we employ the advanced multimodal model, GPT-4 Vision [33], as the evaluator.”
• Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, background): “This combination results in a dataset of 126 million images in total.”
• Will we run out of data? Limits of LLM scaling based on human-generated data (2022, methodology): “For instance, current frontier models like GPT-4V are trained on both image and text data (OpenAI, 2023; Pichai & Hassabis, 2023).”
• Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “For instance, GPT-4 has 1.76 trillion parameters and the model size is 45 GB [63], posing a heavy burden on network storage capacities.”
• GPT-4o System Card (2024, methodology): “This red teaming effort builds upon prior work, including as described in the GPT-4 System Card[6] and the GPT-4(V) System Card[7].”
• A Survey on Knowledge Distillation of Large Language Models (2024, background): “A significant drawback is their limited accessibility and higher cost (OpenAI et al., 2023).”
• DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, methodology): “Figure 6 | Human evaluation results on InternLM-XComposer2-VL (Dong et al., 2024), CogVLM (Wang et al., 2023b), DeepSeek-VL and GPT-4V (OpenAI, 2023b).”