Text-to-Image Generation Grounded by Fine-Grained User Attention
By Jing Yu Koh, Jason Baldridge, Honglak Lee, Yinfei Yang. 2020
65 citations recorded by the source corpus.
Localized Narratives [28] is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TRECS, a sequential model that exploits this grounding to generate images. TRECS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approach outperforms existing direct text-to-image generation models on both automatic metrics and human evaluations: overall, its generated images are more photo-realistic and better match descriptions.
Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Andrea Vedaldi
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
In Natural Language Processing
- Alec Radford
- Scott Gray
- Daniel Cohen-Or
- Andreas R. Dengel
- Dani Lischinski
- Ilya Sutskever
- Aditya Ramesh
- Tobias Hinz
In Generative Models
- Daniel Cohen-Or
- Eli Shechtman
- Dani Lischinski
- Or Patashnik
- Zongze Wu
- Jun-Yan Zhu
- Aditya Ramesh
- Mark Chen
Citations in the corpus
- Zero-Shot Text-to-Image Generation (2021, methodology): “…model architecture with modifications like multi-scale generators (Zhang et al., 2017; 2018), integrating attention and auxiliary losses (Xu et al., 2018), and leveraging additional sources of conditioning information beyond just text (Reed et al., 2016a; Li et al., 2019; Koh et al., 2021).”
- StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Additional supervision was used in other works [37, 25, 20] to further improve the image quality.”
- Adversarial Text-to-Image Synthesis: A Review (2021, methodology): “Given multiple descriptions and their corresponding mouse traces, TRECS retrieves semantic masks from which the images are generated.”