autoRetouch - The way to full automation of Ghost Mannequin Effect (with GANs)

In the field of eCommerce, product images are the only point of contact between a customer and the product. For this reason, the images must create a perfect impression on the customer. This is especially true when it comes to the fashion industry, where the appearance of a product is one of the strongest driving factors for purchase decisions.

For this reason, image processing can be considered a key element of eCommerce for the fashion sector. High quality standards while maintaining high efficiency and reducing costs at the same time is therefore a major challenge. For this purpose, autoRetouch has developed a platform that contributes to automated image processing, where users can pre-define their own workflows to benefit from the AI capabilities of the software. This is the first time that you have the possibility to edit your images with minimal effort and without repetitive workflows.

One of the problems of fashion e-commerce is that a simple photo of the clothing item is not enough. Customers also want to get a taste of how their favorite piece of clothing will fit. Unless the clothing is to be worn by models, which can distract attention from the item itself, at least the contours of a body should still be recognizable. In the so-called “Ghost Mannequin Product Pictures”, only the clothing is visible and it seems to be worn by a ghost.

This kind of representation seems to be very effective in triggering purchase decisions. Unfortunately, the manual process of creating the images is complex and time consuming. autoRetouch has developed an optimized workflow especially for this purpose, reducing the effort of post-processing to the joining of two images. Here in their blog the process is explained in more detail.

What was the way there?

We tried to completely automate the process of developing Ghost Mannequin Product Pictures, i.e. to develop an AI that can combine two images in such a way that the resulting picture looks realistic afterwards.

We got to the root of the matter and faced a number of exciting challenges:

• We worked with a large amount of ultra-high-resolution images, which were characterized by very fine details, such as patterns or brand logos;
• There was no established technology to solve this kind of problem;

• The data available was incomplete: we knew the input, the target, but not the transformations in between.

• We had no guarantee for the success of the platform and therefore it was necessary to manage the project appropriately, both within our team and with the customer.

In summary: A project of great demand, full of uncertainties, which requires a lot of experimentation and a mature management and communication strategy.

How did we approach it?

In data science, it can always happen that projects do not work out the way you want them to. This is not necessarily because you are on the wrong track, as implementation is often difficult and the obvious way is not necessarily the right one. This is especially true when it comes to unexplored areas in advanced technology.

This is what we have focused on from the very beginning:

• To create a setting in which various experiments could be carried out quickly (flexible data pipeline, monitoring of educational processes, scaling of computing power).
• Frequent and honest communication with the autoRetouch team to keep them up to date and to keep their suggestions and expectations in mind was essential for the successful implementation.

Transparency has a very special value in a project with a lot of variables. During development it was therefore essential to discuss with the autoRetouch project team and to communicate possible failures. We were lucky to have colleagues with an affinity for data who knew exactly how the system works.

How does the finished technical solution look like?

In our opinion, Generative Adversarial Networks (short GANs) provided the right framework to address the problem. We identified the pix2pix architecture as a good method for our challenge, because we wanted to develop a network that learns how to fill the missing parts in the front image with information from the image from the inside out. In practice, we were coaching two networks simultaneously:

• A generator that creates the filled images, with the goal of achieving the most realistic result as possible;
• A discriminator, which observes the results of the generator and tries to separate fakes from real images.
• We implemented the entire model - including the pre-processing pipeline - in Tensorflow and trained it on the Google AI platform.

What are the results?

Although the project was not a one-way process, but rather an experimental campaign with an uncertain outcome, we achieved a lot within a month.

Now we can:

• create realistic ghosted pictures. This concerns for example the outlines, patterns and shadows of the pictures, but some fine details like the logos and labels of the brands are still missing.
• Identify major sources of error (inconsistency between input and target images) and set a roadmap for further experiments to solve such problems.

In this way, our customer was also able to get an idea of the actual effort required to solve the problem within the required quality standards and to adjust his own product roadmap accordingly.