FaceShop: Deep Sketch-based Face Image Editing

Tiziano Portenier1 Qiyang Hu1 Attila Szabó1 Siavash Arjomand Bigdeli1
Paolo Favaro1 Matthias Zwicker2
1University of Bern 2University of Maryland
Teaser image
We present a novel system for sketch-based face image editing, enabling users to edit images intuitively by sketching a few strokes on a region of interest. Our interface features tools to express a desired image manipulation by providing both geometry and color constraints as user-drawn strokes. As an alternative to the direct user input, our proposed system naturally supports a copy-paste mode, which allows users to edit a given image region by using parts of another exemplar image without the need of hand-drawn sketching at all. The proposed interface runs in real-time and facilitates an interactive and iterative workflow to quickly express the intended edits. Our system is based on a novel sketch domain and a convolutional neural network trained end-to-end to automatically learn to render image regions corresponding to the input strokes. To achieve high quality and semantically consistent results we train our neural network on two simultaneous tasks, namely image completion and image translation. To the best of our knowledge, we are the first to combine these two tasks in a unified framework for interactive image editing. Our results show that the proposed sketch domain, network architecture, and training procedure generalize well to real user input and enable high quality synthesis results without additional post-processing.
Paper (110MB)
Supplemental (15MB)

SIGGRAPH 2018 Video

Results

Teaser image

Interactive Sessions

Sketching

sketch results

Sketching & Coloring

color results

Smart Copy-Paste

copy-paste results

Comparisons

Globally and Locally Consistent Image Completion

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and Locally Consistent Image Completion. ACM Transactions on Graphics (Proc. of SIGGRAPH 2017) 36, 4, Article 107 (2017), 107:1–107:14 pages.

comparison iizuka

Image-to-Image Translation with Conditional Adversarial Nets

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-To-Image Translation With Conditional Adversarial Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

comparison isola

Eye In-painting with Exemplar Generative Adversarial Networks (ExGANs)

Brian Dolhansky and Cristian Canton Ferrer. 2017. Eye In-Painting with Exemplar Generative Adversarial Networks. arXiv preprint arXiv:1712.03999 (2017).

comparison isola