Faceshop consists of a web-based user interface (left) that enables a user to specify a region to be edited. Moreover, the user can draw strokes and color scribbles to guide the editing process. The core of the back end (right) is a deep image completion network that takes the user input and the original image to synthesize the edited result. Credit: Tiziano Protenier Across popular social media platforms, users are posting countless images every day. On Instagram alone, there are more than 40 billion uploaded photos—a figure that's skyrocketing by 95 million daily. This presents a clear need for intuitive yet robust photo-editing tools that allow the average user to perform advanced editing functions.
And while there is a definite need for interactive image editing with respect to social media, improved editing tools and systems also remain an important aspect of computer graphics and computer vision. However, there is a lack of tools that feature more complex editing functions for inexperienced users, such as changing the facial expression in a photo.
A research team, led by computer scientists from the University of Bern-Switzerland and University of Maryland-College Park, have devised a sketch-based editing framework that enables a user to edit their photos by "sketching" a few strokes on top of them. Their system, called FaceShop, also offers a copy-paste function, which allows users to edit any part of a photo by copying-and-pasting the portion to be edited from another (better) photo, eliminating the need to hand-draw or sketch anything at all.
The team's approach is built on machine-learning techniques, which, in the end, give users more control over their desired edits in real-time and produce more realistic results.
"Most other approaches rely on more traditional, handcrafted techniques, which impose some limitations. For instance, these systems are either [by design] restricted to limited sets of predefined editing operations, or they are very flexible but hard to use and require experienced users to spend a considerable amount of time to perform rather basic edits," says Tiziano Protenier, lead author of the work and Ph.D. candidate at the University of Bern. "In contrast, our system is very flexible and allows untrained users to perform complex edits within minutes using an intuitive interface."
Faceshop consists of a web-based user interface (left) that enables a user to specify a region to be edited. Moreover, the user can draw strokes and color scribbles to guide the editing process. The core of the back end (right) is a deep image completion network that takes the user input and the original image to synthesize the edited result. Credit: Tiziano Protenier Protenier developed the new system with collaborators Qiyang Hu, Attila Szabó, Siavash Arjomand Bigdeli and Paolo Favaro from University of Bern, and Matthias Zwicker from University of Maryland. The team will present their work at SIGGRAPH 2018, held 12-16 August in Vancouver, British Columbia. This annual gathering showcases the world's leading professionals, academics, and creative minds at the forefront of computer graphics and interactive techniques.
In their paper, "FaceShop: Deep Sketch-based Face Image Editing," the researchers demonstrate their new system via several examples. In one, a woman's nose is slightly manipulated, and a strand of hair is edited so that it is pushed away from her face, removing a dark shadow that had appeared on just one side of her face in the original photo. Another photo shows how a user is able to enhance a woman's eye make-up and bring out the color of her eyes. The examples highlighted in the paper showcase how the novel system works, intuitively, and produces high-quality, realistic results.
The team's method is based on generative adversarial neural networks (GANs), a form of artificial intelligence (AI) that, in recent years, has attracted a lot of research interest for its ability to generate realistic looking images. These GANs consist of two AIs that fight against each other. The first component tries to distinguish the generated images from genuine images, whereas the second component tries to produce images that fool the other AI. During training, the two components learn from each other, eventually resulting in a system that autonomously learned to produce realistic looking images, without any human judgement in the loop.
Unlike other AI-driven image editing methods, once the system is trained, users are more in control of the edits to their photos. "Our system features an optimum degree of control, which makes it very interesting from an application point of view," notes Portenier. "Other techniques that try to incorporate user control are either limited in the scope of possible edits or have been shown to work well on the data that was used to train the systems, but fail to produce convincing results with real user input. We are presenting a technique that mitigates this issue, resulting in a system that works surprisingly well in practice."
In future work, the researchers intend to explore additional user interaction tools to add to their framework, and consider how to leverage AI for sketch-based editing of videos.
Explore further: Instagram vamps up photo editing tools
More information: s2018.siggraph.org/
