OtterHD: A High-Resolution Multi-modality Model

By Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, Ziwei Liu. 2023

87 citations recorded by the source corpus.

In this paper, we present OtterHD-8B, an innovative multimodal model evolved from Fuyu-8B, specifically engineered to interpret high-resolution visual inputs with granular precision. Unlike conventional models that are constrained by fixed-size vision encoders, OtterHD-8B boasts the ability to handle flexible input dimensions, ensuring its versatility across various inference requirements. Alongside this model, we introduce MagnifierBench, an evaluation framework designed to scrutinize models' ability to discern minute details and spatial relationships of small objects. Our comparative analysis reveals that while current leading models falter on this benchmark, OtterHD-8B, particularly when directly processing high-resolution inputs, outperforms its counterparts by a substantial margin. The findings illuminate the structural variances in visual information processing among different models and the influence that the vision encoders' pre-training resolution disparities have on model effectiveness within such benchmarks. Our study highlights the critical role of flexibility and high-resolution input capabilities in large multimodal models and also exemplifies the potential inherent in the Fuyu architecture's simplicity for handling complex visual data.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, neutral): “Dealing with higher resolution with grids in LMM are studied in con-current works [1, 27, 52].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, unknown): “For instance, LLaVA-Next [11] and Otter-HD [12] are proposed to enhance the ability based on previous work [7, 13] by improving the image resolution.”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, unknown): “Currently, there are two common technical routes: one involves designing a dual-branch image encoder [32, 72, 73, 116], and the other involves dividing a high-resolution image into many low-resolution tiles [33, 46, 52, 53, 60, 64, 121, 122].”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, background): “Another trend in recent VLMs is increasing input image resolution with the hope of capturing fine-grained details that improve downstream performance (Liu et al., 2023b; Li et al., 2023a).”