It’s no secret that we’re pretty big fans of machine learning and we love thinking of new and exciting ways to use it in Pixelmator Pro. Our latest ML-powered feature is called ML Super Resolution, released in today’s update, and it makes it possible to increase the resolution of images while keeping them stunningly sharp and detailed. Yes, zooming and enhancing images like they do in all those cheesy police dramas is now a reality!
Let’s see some examples
Before we get into the nitty-gritty technical stuff, let’s get right to the point and take a look at some examples of what ML Super Resolution can do. Until now, if you had opened up the Image menu and chosen Image Size, you would’ve found three image scaling algorithms — Bilinear, Lanczos (lan-tsosh, for anyone curious), and Nearest Neighbor, so we’ll compare our new algorithm to those three.
Note that the images below are zoomed in to 200% to make the changes easier to see, but you can zoom out to 100% by clicking the magnifying glass button.
Pretty incredible, right? Until now, if an image was too small to be used at its original resolution, either on the web or in print, there was no way to scale it up without introducing visible image defects like pixelation, blurriness, or ringing artifacts. Now, with ML Super Resolution, scaling up an image to three times its original resolution is no problem at all.
How does it all work?
As computers get ever more powerful, the additional power opens up new possibilities. One of the uses of machine learning, on a very fundamental level, is to make predictions about things. In this case, we gathered a set of images, scaled them down, and then ‘taught’ the algorithm to go from the scaled-down version to the original resolution, high-quality image, predicting the values of each new pixel. The algorithm can’t recreate detail that is too small to be visible but it can make amazing predictions about edges, shapes, contours, and patterns that traditional algorithms simply cannot.
Traditional approaches use (relatively) simple mathematics to interpolate the values of pixels when scaling images.
When adding new pixels, the most basic algorithm, Nearest Neighbor, simply takes the color of the closest neighboring pixel. This results in the classic blocky appearance because the previously imperceptibly small pixels are now big enough to be seen.
The Bilinear algorithm is a little more advanced. A texture map of the image is created according to an algorithm and the values of the 4 closest texels (texture elements) are used when recreating each pixel in the new image. The goal of this approach is to make the transition between pixels much smoother. However, when upscaling quite significantly (or upscaling small images) this algorithm creates the familiar blurry appearance.
Lanczos is yet more advanced, using a complicated mathematical formula to interpolate (another word for predict) the value of any newly created pixels while keeping edges as sharp as possible. Its main disadvantage is that, in its attempts to retain sharpness, the algorithm can sometimes create ringing artifacts. So, ultimately, it’s useful in certain specialized situations, but not much more.
The machine learning way
So, how does the machine learning approach work? Put simply, it takes into account the actual content of every image, attempting to recognize edges, patterns, and textures, recreating detail based on our dataset and extensive training. When upscaling, it can make much better predictions because a red pixel next to a blue pixel can be a completely different type of texture or edge in different images even though, to the primitive approaches, they’re always the same.
The ML Super Resolution convolutional neural network
To create the ML Super Resolution feature, we used a convolutional neural network. This type of deep neural network reduces raster images and their complex inter-pixel dependencies into a form that is easier to process (i.e. requires less computation) without losing important features (edges, patterns, colors, textures, gradients, and so on). The ML Super Resolution network includes 29 convolutional layers which scan the image and create an over-100-channel-deep version of it that contains a range of identified features. This is then upscaled, post-processed and turned back into a raster image. Below is a simplified representation of the neural network.
First, the input image is passed through a high pass filter for basic edge detection. Then, the first convolutional layer reduces the size of these features and pools the data. In the Descriptor Fusion block, the image is scanned to find any JPEG compression blocks within it and this is fused with the other features identified so far.
The next convolutional layers and residual blocks are where the magic happens — these detect the features (edges, patterns, colors, textures, gradients, and so on) in the image, building them up into a complex representation that is over 100 channels deep. In a convolutional neural network, more layers mean better accuracy but with a large enough number of layers, a network becomes near-impossible to train. Residual blocks are designed to increase the complexity and accuracy of networks without making them impossible to train.
Finally, all the features identified by the neural network are enlarged in the Enlarge block. After this, the two residual blocks and the final convolutional layer post-process the data and turn the features back into an image. It’s also important to note that all this happens on-device and the entire trained machine learning model is included inside the Pixelmator Pro app package.
Dealing with noise and artifacts
Small images often contain compression artifacts and noise. If we want our upscaled images to be usable, artifacts and noise shouldn’t be scaled up together with the actual contents of the image. In fact, if possible, they should be removed altogether. And, as mentioned above, ML Super Resolution is designed to do just that, borrowing some of the technologies we developed for ML Denoise to remove both camera noise and JPEG compression artifacts. By the way, in this update, ML Denoise has also been improved, bringing noise removal that is between 2 to 4 times better than before.
Naturally, the machine learning way requires a lot more processing power than the primitive approaches — between 8 to 62 thousand times more, in fact.
Total Floating Point Operations (FLOPs) per pixel*
ML Super Resolution
1,128,062 FLOPs (1.1 megaFLOPs)
* When upscaling 1 pixel by 300%, creating 9 pixels.
Making this available in an app like Pixelmator Pro has only become possible in the last couple of years — even on Mac computers from 5 or so years ago, ML Super Resolution can take minutes to process a single image due to slower performance and less available memory. On the latest hardware, however, images are processing in a few seconds, and even faster on iMac Pro, Mac Pro, or any Mac with multiple GPUs thanks to our use of Core ML 3 and its multi-GPU support. For the same reasons, the performance of ML Super Resolution is also significantly improved when using an eGPU.
Compute time with eGPU2
MacBook Pro (13-inch, Mid 2012)
MacBook Air (13-inch, 2018)
MacBook Pro (16-inch, 2019)
iMac Pro (2017)
1. For this test, a 300,000 pixel image was upscaled to three times its original size.
2. Tested using an AMD Radeon RX 5700 XT eGPU.
3. External GPUs require a Thunderbolt 3-equipped Mac.
Using the 2012 MacBook Pro as a baseline, the latest devices are up to 200x faster!
We’re incredibly excited about ML Super Resolution and we honestly hope you’re going to love it too. If you’d like to, you can download all the images in this blog post using the link below and test everything in today’s update out for yourself.
Pixelmator Pro 1.5.4 is now available from the Mac App Store, so head on down there and make sure you’re up to date. The trial version has also been updated so if you don’t yet have a copy, you’re welcome to try it out. That’s it for now, but we hope to surprise you with one more cool new feature before the year is up — stay tuned!
ML Super Resolution requires macOS Mojave or later.
Did you know the original Pixelmator turned 12 years old a couple of months ago? Crazy! The fact that lots of people still love and use the app every day is a real testament to its quality. While Pixelmator Pro is its successor (Pixelmator 2, if you will), we’re planning to support the original Pixelmator for a while longer. And today, we’ve released Pixelmator 3.9 (codenamed Classic) with macOS Catalina support, including support for Sidecar and Apple Pencil.
Along with some optimizations for macOS Catalina and a few fixes, we’ve also added support for Sidecar, so you can extend your Pixelmator workspace using your iPad as a second display. And with support for Apple Pencil, you can paint, retouch, and illustrate with ultimate precision!
This free update is out now on the Mac App Store, so head on down there and make sure you’re all up to date.
Good news, everyone! We’ve just shipped a smaller Pixelmator Pro update with a big new feature called Soft Proof Colors. This one’s for all you print designers out there.
Soft proofing lets you see what an image would look like when reproduced on a different output device — for example on a different monitor or when printed. And for most Pixelmator Pro users, the most important thing is being able to soft proof images with CMYK color profiles when working on designs that will be printed. As Pixelmator Pro already had the ability to convert to CMYK when exporting, with soft proofing, it becomes an even more powerful image editor for those of you creating designs for print.
A few days ago, we also started the Pixelmator Pro Upgrade Sale to encourage any original Pixelmator users to give Pixelmator Pro a chance and it’s been a great couple of days in many different ways! More on that later, but the sale ends on Tuesday, October 29th, so you have a few more days to take advantage of it — it really is a fantastic offer.
While we get back to working on Pixelmator Pro 2.0 (oh yes, we’ve already started and it’s going to be an amazing free upgrade), feel free to check out the full release notes for this update by visiting our What’s New page or just go ahead and view Pixelmator Pro 1.5.1 on the Mac App Store.
We’ve said (quite a few times now) that Pixelmator Pro is Pixelmator v2 and the future of Pixelmator. It’s not a more ‘pro-oriented’ app — it’s pro image editing for everyone.
And we think Pixelmator Pro is an incredible app that every user of the original Pixelmator would absolutely love using. That’s one of the reasons why we recently created the upgrade bundle on the Mac App Store and an upgrade page comparing the two apps. Though for those of you who bought Pixelmator on sale (and quite a few did), the discount was very small or, in certain regions, there was no discount at all.
For that reason, today, we’re starting a limited-time upgrade sale letting everyone get Pixelmator Pro for just $19.99. This is much lower than our usual sale price, so you’d be crazy not to take advantage of it!
Edit: The sale has now ended and Pixelmator Pro is back to its usual price of $39.99.
Pixelmator Pro just got an awesome major update! Version 1.5 (codenamed Avalon) brings full support for macOS Catalina, the upcoming Mac Pro and Pro Display XDR, introduces intelligent, machine-learning powered noise removal, big performance improvements, and a whole lot more. Here’s a quick rundown of what we’ve added.
macOS Catalina Support
Pixelmator Pro is now fully compatible with macOS Catalina, including support for Sidecar and Apple Pencil, which means you can extend your desktop workspace using your iPad as a second Mac display! And with Apple Pencil support, you can paint, sketch, create graphic designs, and retouch your photos with pressure-sensitivity, acceleration, and tilt support as well as support for the double-tap gesture. Plus, as Pixelmator Pro fully supports Touch Bar, handy controls appear at the bottom of your iPad when using Sidecar — even if your Mac doesn’t have a Touch Bar.
Thanks to its Metal-based architecture and GPU-powered editing engine, Pixelmator Pro absolutely flies on the upcoming Mac Pro. We’ve been able to optimize multi-GPU support so that with multiple graphics processors in Mac Pro, you’ll see big performance boosts in Pixelmator Pro. And performance increases even more with each additional GPU. For example, Pixelmator Pro applies the new Core ML-powered ML Denoise up to 2.5x faster and effects up to 2x faster on Mac Pro with 2 GPUs compared to iMac Pro.
We’ve also added support for the upcoming Pro Display XDR. The display’s true 10-bit color depth and P3 wide color gamut are natively supported in Pixelmator Pro. And the new Extended Dynamic Range Mode lets you display clipped details in RAW photos. To do this, we use the display’s 1600-nit peak brightness to bring out details in the highlights without compressing the dynamic range of the rest of the photo, bringing a never-before-possible RAW editing and viewing experience.
ML Denoise is an amazing machine learning-powered noise removal tool. Integrated via Core ML, ML Denoise effortlessly removes luminance and color noise created by cameras in low-light photos. What’s more, it can even reduce artifacts caused by image compression algorithms, improving image quality while preserving important details!
Pixelmator Pro now has an end-to-end Metal pipeline for rendering and editing and a new asynchronous zoom engine, which brings some big performance improvements. Zooming and scrolling is now at least 10x faster and always responsive, effects are up to 2.7x faster, and painting is up to 2.4x faster!
SF Symbols, the new set of over 1,500 configurable vector icons designed by Apple, is fully supported in Pixelmator Pro. So you can easily open SF Symbols templates, customize them to create your own symbols, and even drag and drop symbols right into existing documents!
This fantastic fifth major update to Pixelmator Pro is free for all existing users, so download it, try it, and let us know how we did. As you might guess, we’re already working on some more really incredible additions to Pixelmator Pro and we can’t wait to share them with you. Until next time!
The Pixelmator Photo 1.1 major update is here! And it really is a big one. We’ve added full compatibility with iPadOS 13, full-featured batch editing, an all-new and improved workflow, export resizing, and some smaller improvements and fixes.
Everything seems really cool in its own right so it’s difficult to pick just one headline feature but let’s start with iPadOS. iPad OS 13 support is a very big part of this update and with our Files-based design, you can now take full advantage of support for external drives and new external locations.
Batch editing is huge! We’ve brought a really full-featured batch editing experience to iPad that’s, let’s face it, even better than what we currently have on the Mac with Pixelmator Pro. A lot of that is down to Pixelmator Photo being a dedicated photo editor, of course, but it’s amazing to see all the incredible machine learning features now being available for batch editing on iPad.
The one new feature that might not be as flashy but will directly affect every single current and future Pixelmator Photo user is the all-new workflow and direct integration with your iCloud Photos library. Gone are the days of having to import photos and manage separate Pixelmator Photo files, everything is now simple and intuitive.If you’re editing in your Photos library, edits are automatically saved to the same images you open. Nondestructive edits are preserved too! And if you’re editing in Files, Pixelmator Photo does some magic to save changes back to the same image while preserving nondestructive edits in a linked file.
Finally, we’ve also added the ability to export images at different sizes, which is a nice little extra that we found time to squeeze in between all the other huge things and foundational changes to Pixelmator Photo.
The time has come. It’s time for us to give all of you still using the original Pixelmator a friendly nudge to check out Pixelmator Pro, the successor to the image editor you know and love. And to do that, we’ve created a special Upgrade page outlining all the goodness of Pixelmator Pro.
The new page includes an extremely in-depth comparison of the two apps, some FAQs about the upgrade, and a little roadmap that even existing Pixelmator Pro users will find very interesting. Naturally, there’s an upgrade discount of up to 50% for all of you who purchased the original Pixelmator from the Mac App Store.
If you think it took us a while to get this page together, you’d definitely be right. But, to be honest, we’ve just been focusing like crazy on Pixelmator Pro itself and making the app as good as it can be. When (nearly two years ago now) we launched Pixelmator Pro, its average worldwide rating was stuck at 3.7 for about a year and that told us there was a whole bunch of things, large and small, that need to be improved. So we focused on doing just that. With lots of free updates (27 and counting) since then, we’re incredibly proud to see the app now rated 4.7 worldwide with over 4,000 ratings and over 3,300 five-star ratings. And we’re also incredibly proud of the Mac App of the Year award it received from Apple as well.
What’s more, when Pixelmator Pro launched, app bundles (and, by extension, upgrade bundles) were not yet available on the Mac App Store. So, now that we can provide an upgrade discount, we think it’s a great time to give you existing Pixelmator users a handy page to answer all the questions you might have about upgrading to Pixelmator Pro and some more information on the app itself.
A few months ago, WWDC 2019 bought some really incredible Mac software and hardware announcements. A brand new, ultra-powerful Mac Pro, the stunning Pro Display XDR, and of course the awesome software news with macOS Catalina and all its new features.
Now, after several months of hard work, we’re almost ready to release a major update to Pixelmator Pro, adding support for all this. And today, we excited to share a public beta version of Pixelmator Pro 1.5 (codenamed Avalon) so you can test it out before the update hits the Mac App Store.
In our blog post after WWDC, we focused mostly on Mac Pro, Pro Display XDR, and Sidecar, but this update has a whole lot more than that. For example, there’s ML Denoise, which removes noise from photos while intelligently preserving details using, you guessed it, advanced machine learning! What’s more, ML Denoise can even remove JPEG compression artifacts. There’s also some major performance improvements, including a completely redesigned and always responsive zoom engine, painting tool improvements, and faster effect rendering. We’ve even squeezed in full support for SF Symbols!
This is going to be another awesome update and if you’d like to get an early look at it before everyone else, all you need to do is sign up for our public beta. Feedback is always welcome at firstname.lastname@example.org!
Exciting news incoming! Pixelmator Photo 1.1 is close to being finished and is shaping up to be a really fantastic first major update, bringing batch editing, an all-new editing workflow, and a whole lot more.
We know library and file management workflows are incredibly important to pretty much every photographer out there, so that’s been our primary focus with Pixelmator Photo 1.1. Thanks to the all-new workflow, you can now enjoy a completely Photos-centric editing and file management experience without having to import photos into Pixelmator Photo. That means no more of those pesky duplicate files cluttering up your iPad. And you can even revert or delete photos in your library right from Pixelmator Photo.
Naturally, for every previously edited photo that you open, you’ll still be able to find your previous edits as we’ll automatically save and link them to the images in your Photos library. What’s more, if you prefer editing images in the Files app, this new workflow magically links edits to images in Files, too.
And you want to know something crazy? That’s probably not even the biggest new feature in this update. We’ve also added seriously powerful, machine learning-powered batch editing — we think you’re going to love it and we’ll tell you a little more about that later.
We’ve really enjoyed hearing your feedback in our previous betas so we’re opening up 300 extra spots for this update. If you’d like to help us out, sign up using TestFlight below. For the rest of you, it shouldn’t be too long before you can get your hands on this incredible update!