In 2017, ex-Huawei algorithm scientist Zhao Weijie and serial entrepreneur Cai Tianyi launched Makaron, an AI-powered photo-editing app to rival Photoshop. Originally known as Versa, the duo's app has taken smartphone photo editing to a whole new level by allowing subjects and backgrounds to be edited separately.
On the Apple App Store page for Makaron, video and photo illustrations show a girl duplicating her image in a photo to make it look like she is having afternoon tea with herself and changing the photo's background to make it appear like it was taken in Paris – editing done entirely on the phone with just a few clicks.
According to Makaron's CEO and co-founder Cai, AI has helped lower the technological barrier to editing photographs, allowing more people to create artworks. “What we want to – and can – do is empower creators and unleash their creativity by providing a tool to help materialize creative ideas,” he said.
Rough start
Inspired by Prisma, another photo-editing app that was launched in June 2016, Makaron similarly uses AI to turn a photo into an artwork by letting users add filters to a photo in one to two seconds. In comparison, it takes the Prisma app 30 to 50 seconds to carry out the same task.
Makaron's speedy performance can be attributed to ConceptNet, its in-house neural network developed by CTO and co-founder Zhao, which lets the app recognize objects in photos quickly and precisely. Unlike conventional, data-driven deep-learning frameworks, ConceptNet cuts down on the time and data needed for algorithm training, making it easier to offer new filters.
“It took around 80,000 pictures and eight hours to train a conventional neural network to transfer the style of one image, while ConceptNet needed only 100 pictures and 40 minutes,” Cai said.
Despite its edge in applying filters more quickly, Makoron failed to achieve the success of Prisma, which saw over 7.5m downloads only one week after its launch. After its release in July 2017, the Makaron app had only 2m downloads by the end of that year. It turned out that the novelty of Prisma and similar apps had faded and was no longer attractive to users in China, revealing how is hard for a photo editing app with only one feature to gain a large user base.
New feature draws more users
In October 2017, Makaron rolled out its second biggest feature, one that eventually shaped the app: object-background segmentation. Although many photo editing apps had been dabbling with automatically segmenting objects from background, Makaron managed to do a better job, which was no mean feat.
Makaron's ConceptNet had a 95.6% accuracy when it comes to image segmentation as compared to the 88% accuracy that Google's Deeplab V3+ attained. ConceptNet could discern over 20 objects, like pets and cars, as the foreground objects. “Our neural network can discern locks of hair from the background. Everyone who has any idea about deep learning knows how hard that is,” said Zhao.
This new feature allows Makaron users to edit foreground objects and the background separately, including moving, enlarging, shrinking, doubling an object; changing an individual object's exposure, contrast, saturation; and even separately applying filters on different objects.
The object-background segmentation feature opens up a whole new world of possibilities for photo editing. Basic applications include cutting unrelated people – photo bombers, for example – out of a picture while subtler applications include making people's faces look brighter without changing the lighting exposure of the background. The feature can also accomplish effects like removing an object from a photo with a click, which previously could only be carried out using computers and professional software, e.g., Photoshop.
In an effort to increase user stickiness, Makaron has built a community platform within its app for users to share the photo-artworks that they have created, effectively turning the startup into a combination of Photoshop and Instagram.
“Our tool can help produce creative works which in turn becomes a draw for users,” said Cai. He is also the co-founder of Gewara, a movie ticket booking startup with over 40m users, and is well versed at retaining users.
Having hit on its product-market fit, Makaron saw a surge in its users, with numbers exceeding 20m by the end of 2018.
Profit through enterprise clients
As its user base continues to enlarge, Makaron remains free to download and use for individual users. It provides APIs to enterprise clients and charges them for this service. Its main customers are community apps that want to incorporate some of Makaron’s functions, such as style transfer, object-background segmentation and smart autofill, into their own platforms as fun features.
Although the startup has not named any of its enterprise clients, it claimed to have provided APIs to some “mainstream” community apps. China's social media giant Tencent, which participated in Makaron's Series A funding in December 2018, has also introduced some clients to Makaron.
Makaron is able to apply its neural network to video rendering, but has not rolled out this feature yet. “We can do style transfer, real-time 3D modeling, 2D/3D real-time pose estimation, but video rendering requires a very high bandwidth which cannot be delivered by a regular smartphone. Phones with AI chips are not prevalent yet,” said Cai.
Makaron has thus opted to work with movie and video game companies by providing them with its customised algorithms. To give the 2017 movie, Loving Vincent, the feel of an oil painting, the director had asked artists to paint all the scenes before turning the paintings into videos, a process that took them four years to complete. Cai said Makaron could have accomplished this effect in 1% of that time using its style transfer technology, adding, “I hope our technologies can be used to shoot a film in the future."
