Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach

By Yahui Liu, Marco De Nadai, Deng Cai, Huayang Li, Xavier Alameda-Pineda, Nicu Sebe, Bruno Lepri. 2020

62 citations recorded by the source corpus.

Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to use richly-annotated image captioning datasets. In this work, we propose a novel unsupervised approach, based on image-to-image translation, that alters the attributes of a given image through a command-like sentence such as "change the hair color to black". Contrarily to state-of-the-art approaches, our model does not require a human-annotated dataset nor a textual description of all the attributes of the desired image, but only those that have to be modified. Our proposed model disentangles the image content from the visual attributes, and it learns to modify the latter using the textual description, before generating a new image from the content and the modified attribute representation. Because text might be inherently ambiguous (blond hair may refer to different shadows of blond, e.g. golden, icy, sandy), our method generates multiple stochastic versions of the same translation. Experiments show that the proposed model achieves promising performances on two large-scale public datasets: CelebA and CUB. We believe our approach will pave the way to new avenues of research combining textual and speech commands with visual attributes.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Some methods [10, 31, 27] use a GAN-based encoderdecoder architecture, to disentangle the semantics of both input images and text descriptions.”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, methodology): “[38] propose a multi-domain and multi-modal method to explicitly model the visual attributes of an image and learn how to translate them through automatically generated commands.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “On the forefront of image manipulation there are also many works that address text-guided image manipulation [174, 175, 176, 177, 178, 179], which might be a more ﬂexible interface for users than, e.g., editing semantic maps or (a limited amount of) labels.”
TediGAN: Text-Guided Diverse Image Generation and Manipulation (2020, methodology): “[31] propose a multi-domain and multimodal method to explicitly model the visual attributes of an image and learn how to translate them through automatically generated commands.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, methodology): “Closest to ours is the TA-GAN [52] scheme that takes the image caption as input to describe attributes for conditional image manipulation, followed by [37] and improved by [47].”