CLIPstyler: Image Style Transfer with a Single Text Condition

By Gihyun Kwon, Jong Chul Ye. 2021

352 citations recorded by the source corpus.

Existing neural style transfer methods require reference style images to transfer texture information of style images to content images. However, in many practical situations, users may not have reference style images but still be inter-ested in transferring styles by just imagining them. In order to deal with such applications, we propose a new framework that enables a style transfer ‘without’ a style image, but only with a text description of the desired style. Using the pre-trained text-image embedding model of CLIP, we demonstrate the modulation of the style of content images only with a single text condition. Specifically, we propose a patch-wise text-image matching loss with multiview augmentations for realistic texture transfer. Extensive experimental results confirmed the successful image style transfer with realistic textures that reflect semantic query texts.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Recent models have leveraged CLIP [48] embeddings to guide image editing using text [5, 9, 14, 29, 32, 42, 45, 71].”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, background): “Kwon and Ye [21], Frans, Soros, and Witkowski [13], Chen, Dumay, and Tang [5], Liu et al.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (2022, background): “While most works that require only text are limited to global editing [14, 31], Bar-Tal et al. [5] proposed a text-based localized editing technique without using masks, showing impressive results.”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “…domain adaptation (Gal et al., 2021; Kim et al., 2022), video manipulation (Tzaban et al., 2022; Bar-Tal et al., 2022), motion synthesis (Tevet et al., 2022; Petrovich et al., 2022), style transfer (Kwon & Ye, 2021; Liu et al., 2022) and even texture synthesis for 3D objects (Michel et al., 2021).”
Inversion-based Style Transfer with Diffusion Models (2022, methodology): “Text guided stylization [13, 15, 26, 34] produces an artistic image from a natural image and a text prompt, but usually the text prompt for target style can only be a rough description of material (e.”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, background): “Several works [11,14,21,28] take a test-time optimization approach and leverage CLIP without using a pre-trained generator.”
Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models (2023, methodology): “…the context of image editing, while most methods are generally limited to global edits [Chefer et al. 2022a; Crowson et al. 2022; Gal et al. 2022b; Kwon and Ye 2022], several works introduce a user-provided mask to specify the region that should be altered [Avrahami et al. 2022a; Bau et al. 2021;…”
Prompt-to-Prompt Image Editing with Cross Attention Control (2022, background): “, no masks) are limited to global editing [9, 23], Bar-Tal et al.”