FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling

By Haoning Wu, Chaofeng Chen, Jingwen Hou, Liang Liao, Annan Wang, Wenxiu Sun, Qiong Yan, Weisi Lin. European Conference on Computer Vision, 2022

327 citations recorded by the source corpus.

Current deep video quality assessment (VQA) methods are usually with high computational costs when evaluating high-resolution videos. This cost hinders them from learning better video-quality-related representations via end-to-end training. Existing approaches typically consider naive sampling to reduce the computational cost, such as resizing and cropping. However, they obviously corrupt quality-related information in videos and are thus not optimal for learning good representations for VQA. Therefore, there is an eager need to design a new quality-retained sampling scheme for VQA. In this paper, we propose Grid Mini-patch Sampling (GMS), which allows consideration of local quality by sampling patches at their raw resolution and covers global quality with contextual relations via mini-patches sampled in uniform grids. These mini-patches are spliced and aligned temporally, named as fragments. We further build the Fragment Attention Network (FANet) specially designed to accommodate fragments as inputs. Consisting of fragments and FANet, the proposed FrAgment Sample Transformer for VQA (FAST-VQA) enables efficient end-to-end deep VQA and learns effective video-quality-related representations. It improves state-of-the-art accuracy by around 10% while reducing 99.5% FLOPs on 1080P high-resolution videos. The newly learned video-quality-related representations can also be transferred into smaller VQA datasets, boosting performance in these scenarios. Extensive experiments show that FAST-VQA has good performance on inputs of various resolutions while retaining high efficiency. We publish our code at https://github.com/timothyhtimothy/FAST-VQA.

View Paper: https://www.semanticscholar.org/paper/9ba3cfc172617c301f914b4835f532313210cc4f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yu Qiao
2. Limin Wang
3. Ziwei Liu
4. Luc Van Gool
5. Yinan He
6. Haoning Wu
7. Chaofeng Chen
8. Yuanhan Zhang

Citations in the corpus

• Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels (2023, background): “On the other hand, the ensemble gain with FAST-VQA suggests that though the Q-A LIGN (1fps) has been effective, since we haven’t yet input all frames of the video into the LMM, there is still room for future improvements.”
• Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives (2022, methodology): “Many studies [13, 59, 68] suggest that technical quality perception should consider global semantics to better assess distortion levels.”
• VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “Meanwhile, the Video Quality Assessment (VQA) methods [63, 94, 108–114] are primarily designed for real videos, thereby neglecting the unique challenges posed by generative models, such as artifacts in synthesized videos.”