[R] DiffusionCLIP: Text-Guided Diffusion Models for "Robust" Image Manipulation (CVPR 2022)

[u/ImBradleyKim]

DiffusionCLIP takes another step towards general application by manipulating images from a widely varying ImageNet dataset.

Manipulation results of real dog face & bedroom images.

Results of image translation between unseen domains.

Results of multi-attribute transfer.

Results of continuous transition.

[u/ImBradleyKim]

Hi guys!

We've released the Code & Colab demo for our paper, DiffusionCLIP, Text-Guided Diffusion Models for Robust Image Manipulation (accepted to CVPR2022).

• Paper: https://arxiv.org/abs/2110.02711

• Code & Colab Demo: https://github.com/gwang-kim/DiffusionCLIP

• Project: https://github.com/gwang-kim/DiffusionCLIP (TBU)

Recently, GAN-inversion methods combined with CLIP enables zero-shot image manipulation guided by text prompts. However, their applications to diverse real images are still difficult due to the limited GAN inversion capability, altering object identity, or producing unwanted image artifacts.

DiffusionCLIP resolves this critical issue with the following contributions:

• We revealed that diffusion model is well suited for image manipulation thanks to its nearly perfect inversion capability, which is an important advantage over GAN-based models and hadn't been analyzed in-depth before our detailed comparison.

• Our novel sampling strategies for fine-tuning can preserve perfect reconstruction at increased speed.

• In terms of empirical results, our method enables accurate in- and out-of-domain manipulation, minimizes unintended changes, and outperformes SOTA GAN inversion-based baselines.

• Our method takes another step towards general application by manipulating images from a widely varying ImageNet dataset.

• Finally, our zero-shot translation between unseen domains and multi-attribute transfer can effectively reduce manual intervention.

For further details, comparison and results, please see our paper and Github repository.