Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels

By Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun and 4 more. International Conference on Machine Learning, 2023

571 citations recorded by the source corpus.

The explosion of visual content available online underscores the requirement for an accurate machine assessor to robustly evaluate scores across diverse types of visual contents. While recent studies have demonstrated the exceptional potentials of large multi-modality models (LMMs) on a wide range of related fields, in this work, we explore how to teach them for visual rating aligned with human opinions. Observing that human raters only learn and judge discrete text-defined levels in subjective studies, we propose to emulate this subjective process and teach LMMs with text-defined rating levels instead of scores. The proposed Q-Align achieves state-of-the-art performance on image quality assessment (IQA), image aesthetic assessment (IAA), as well as video quality assessment (VQA) tasks under the original LMM structure. With the syllabus, we further unify the three tasks into one model, termed the OneAlign. In our experiments, we demonstrate the advantage of the discrete-level-based syllabus over direct-score-based variants for LMMs. Our code and the pre-trained weights are released at https://github.com/Q-Future/Q-Align.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/30cc93dff85ed7e78e4df3b609393ea8da6cc6b6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Peiran Ren
2. Lei Zhang
3. Tao Yang
4. Xuansong Xie
5. Dongxu Li
6. Haoning Wu
7. Yu Qiao
8. Luc Van Gool

Citations in the corpus

• Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, unknown): “These popular methods make use of large language models to decompose compositional text prompts into simpler question-answer pairs for analysis, e.g., Llama-2 for TIFA, FlanT5 for VQ2, and ChatGPT for Davidsonian.”
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, methodology): “Specifically, PLLaVA, VideoChat2, VideoLLaVA and ShareGPT4Video [Xu et al., 2024, Lin et al., 2023, Chen et al., 2023, 2024] are fed with 16 frames, while LLaVA-Next-Video (34B/M7B) [Zhang et al., 2024] are fed with 32 frames.”
• Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization (2023, methodology): “More visual re-sults can be found in the supplementary material .”