HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models

By Sharon Zhou, Mitchell L. Gordon, Ranjay Krishna, Austin Narcomey, Li Fei-Fei, Michael S. Bernstein. 2019

136 citations recorded by the source corpus.

Generative models often use human evaluations to measure the perceived quality of their outputs. Automated metrics are noisy indirect proxies, because they rely on heuristics or pretrained embeddings. However, up until now, direct human evaluation strategies have been ad-hoc, neither standardized nor validated. Our work establishes a gold standard human benchmark for generative realism. We construct Human eYe Perceptual Evaluation (HYPE) a human benchmark that is (1) grounded in psychophysics research in perception, (2) reliable across different sets of randomly sampled outputs from a model, (3) able to produce separable model performances, and (4) efficient in cost and time. We introduce two variants: one that measures visual perception under adaptive time constraints to determine the threshold at which a model's outputs appear real (e.g. 250ms), and the other a less expensive variant that measures human error rate on fake and real images sans time constraints. We test HYPE across six state-of-the-art generative adversarial networks and two sampling techniques on conditional and unconditional image generation using four datasets: CelebA, FFHQ, CIFAR-10, and ImageNet. We find that HYPE can track model improvements across training epochs, and we confirm via bootstrap sampling that HYPE rankings are consistent and replicable.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

In Evaluation & Benchmarks

Citations in the corpus

Improved Techniques for Training Score-Based Generative Models (2020, methodology): “Metrics: We use FID [11] and HYPE ∞ [16] scores for quantitative comparison of results.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, background): “On the other hand, unconditional image models can synthesize photorealistic images (Brock et al., 2018; Karras et al., 2019a;b; Razavi et al., 2019), sometimes with enough ﬁdelity that humans can’t distinguish them from real images (Zhou et al., 2019).”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “One idea is to conduct user studies on cloud-sourcing platforms for either assessing the samples’ image quality [17, 64, 82] or identifying duplicate images [2].”
Adversarial score matching and improved sampling for image generation (2020, background): “To resolve that matter, they proposed the use of a new metric (Zhou et al., 2019) that did not have this issue.”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, background): “For example, HYPE (Zhou et al., 2019) is an evaluation platform for image generation, ChatEval (Sedoc et al., 2019) is an evaluation platform for chatbots, and more recently, Zellers et al. (2020) present a leaderboard for the advice generation task introduced in their work.”
TuringAdvice: A Generative and Dynamic Evaluation of Language Use (2021, methodology): “An alternative strategy requires submitters to pay the Mechanical Turk fees themselves; this model was used for the HYPE leaderboard in computer vision (Zhou et al., 2019).”
Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Similar to HYPE [171] which standardized user evaluation studies for image quality, the community could beneﬁt from a standardized user evaluation strategy for image-text alignment.”
Detecting CNN-Generated Facial Images in Real-World Scenarios (2020, background): “Moreover, it differs from [54] since we do not include any time constraints or training phase and evaluate more logical image resolutions.”
On Aliased Resizing and Surprising Subtleties in GAN Evaluation (2022, background): “One idea is to conduct user studies on cloud-sourcing platforms for either assessing the samples’ image quality [13, 59, 76] or identifying duplicate images [1].”
Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis (2023, methodology): “Similar methodology is used in HYPE [38] for evaluating generated human faces.”
Evaluating Machines by their Real-World Language Use (2020, methodology): “12This model is used for the HYPE leaderboard in computer vision (Zhou et al., 2019).”
Improved Natural Language Generation via Loss Truncation (2020, background): “Distin-guishability has recently gained attention as a way to learn and evaluate models based on both sample quality and diversity (Hashimoto et al., 2019; Zhou et al., 2019; Zellers et al., 2019; Gehrmann et al., 2019).”
The Role of ImageNet Classes in Fréchet Inception Distance (2022, background): “As an alternative to these automatic metrics, a human-in-the-loop evaluation of model quality has been described [61].”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “Zhou et al. [166] defined HYPE (human eye eerceptual evaluation) as a human benchmark for generative realism.”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, background): “Figure 13: HYPE tests generative models for how realistic their images look to the human eye [51].”