June 7, 2018
The new machine learning-powered Auto White Balance
Pixelmator Pro includes one of the best collections of color adjustment tools in any app on any device. And we want to make them even better. In Pixelmator Pro 1.1 Monsoon, we added some super cool Auto buttons next to three color adjustments: White Balance, Lightness, and Hue & Saturation. Although those buttons may be small, there’s a lot of machine learning power behind them and we wanted to talk a little bit about one in particular — Auto White Balance.
Fixing white balance
When editing photos, fixing the white balance is almost always the first thing photographers do before moving on to any other edits. It’s important to get it right because all your later edits depend on it. And if you’re not used to doing it, it’s not all that easy — there are two different sliders that are influenced by each other and it’s not that obvious what they do. I mean, adding yellow cancels out a blue tint, how does that work?
So, in Pixelmator Pro 1.1 Monsoon, we’ve added the machine learning-powered Auto White Balance to make fixing the white balance in any photo as simple as clicking one little button. Behind that button, there’s a 200-layer-deep neural network trained on millions of image pairs and integrated via the awesome Core ML.
Did you know that neural networks are practically magic and no one knows how they really work? Basically, getting machine learning models to do what you want them to requires you to test them on really huge datasets. The models then make complicated, seemingly random connections based on your tests and their quality depends on how good your tests and datasets are.
In this case, we needed millions of images with bad white balance and good white balance. It probably goes without saying that there’s no giant repository of badly white balanced images to just download. Even if there was, we’d still need to get someone to fix each one and that’s not exactly efficient either. For that reason, we had to find a different approach. We decided to take millions of good images and create a separate algorithm to ‘break’ them in as natural a way as possible to mimic real-world white balance issues.
Neural network training
Once we had our image pairs, we could start training. Except there was another little problem. Most neural networks work with small, fixed-resolution images, which usually have to be square. That meant we needed to turn every photo into a uniform small square image. But if the photo is portrait or landscape, potentially important details at the edges of the image would be lost if we cropped them. If we simply stretched them, then they wouldn’t look like actual photos people edit.
To get around this, we took each image and applied content-aware scaling. This separate algorithm identified features in the original images (for example, a sky, some trees, or people) and scaled them according to their importance to the photo. The content-aware scaling makes for some pretty strange-looking images — people look a little like weird aliens — but the neural network likes them a little better. Now we could start training and getting our model to learn. And once that was done, we could start testing how accurate our approach was.
Quality and accuracy
Most image editing apps have some sort of non-machine learning automatic white balance correction tools and we compared our approach with the results you get in those apps. We measured quality using the industry standard CIEDE2000 metric and found our Auto White Balance tool to be at least 50% more accurate, and usually between 2 to 10 times better than those standard tools. And we think our approach passes the eye test even better. Just looking at the results, the Auto White Balance adjustment seems to be much more natural and visually pleasing.
One awesome feature of Auto White Balance is support for RAW images and any extended data within them. Our trained neural network uses extended-range information to correct white balance issues in RAW images with improved accuracy. Sweet!
Here are the Auto Color Adjustments getting a mention during the Platforms State of the Union at this year’s Worldwide Developer Conference in San Jose.
Better than humans
To round things off, here’s a fun fact. The new Auto White Balance correctly fixes the white balance in the infamous “dress” photo, which became an internet phenomenon in 2015. And that photo isn’t even in the training dataset! This is an especially good example of white balance and lighting issues which are tricky even for humans to solve. And Pixelmator Pro now makes it effortless with the help of breakthrough machine learning.
If you want to try out these awesome machine learning features on the dress or any of your own photos, you can download Pixelmator Pro 1.1 Monsoon — it’s available now from the Mac App Store and it’s a free update for everyone who has already bought Pixelmator Pro.