All about the new ML Super Resolution feature in Pixelmator Pro

[Andrius]

It’s no secret that we’re pretty big fans of machine learning and we love thinking of new and exciting ways to use it in Pixelmator Pro. Our latest ML-powered feature is called ML Super Resolution, released in today’s update, and it makes it possible to increase the resolution of images while keeping them stunningly sharp and detailed. Yes, zooming and enhancing images like they do in all those cheesy police dramas is now a reality!



Read the full blog entry here.

[st3f]

like they do in all those cheesy police dramas

I think that Blade Runner beat them to it by a decade or two. There's also a film where the protagonist's face is on an unclear photograph from a murder scene. The image is being 'enhanced'. He only has a few hours of freedom to clear his name before the program completes and makes him the prime suspect. I wish I could remember the film.
I'm looking forward to using this... but for more mundane purposes, though.

[Andrius]

By the way, it looks like the update is taking a lot longer than usual to appear in the Mac App Store — but it's coming. In the meantime, I'll leave this little gem here for your viewing pleasure.

[ChrisRosa]

I'm not seeing the ML Super Resolution option in the Image menu. I'm on a 2017 13-inch MBP 4TB3. Any ideas?

[gvisser]

Thanks Pixelmator!
This is a feature I've been wanting for years!

st3f The movie you are thinking of is "No Way Out" with Kevin Costner.

[Michael Long]

I think that Blade Runner beat them to it by a decade or two. There's also a film where the protagonist's face is on an unclear photograph from a murder scene. The image is being 'enhanced'. He only has a few hours of freedom to clear his name before the program completes and makes him the prime suspect. I wish I could remember the film.
I'm looking forward to using this... but for more mundane purposes, though.

No Way Out. Kevin Costner, Sean Young.

[st3f]

st3f The movie you are thinking of is "No Way Out" with Kevin Costner.

No Way Out. Kevin Costner, Sean Young.

Now... the question is... should I
Option A
1. Track down the film.
2. Work out when it's set and how long the rendering takes in film.
3. Estimate the pixel dimension and the quality of the rendering hardware used.
4. Work out how long it takes today on current equivalent hardware.
5. Plot some Moore's law graph with two data points and a lot of supposition.

Option B
Let sleeping dogs lie as I seem to recall the film was a bit of a stinker.

I've found it to rent on Amazon Prime for £3.49. Is it worth £3.49 and 1h54m of my life to pursue Option A? Hmmm... I'm going to watch the trailer and make my mind up after that.

Thanks for the responses, gvisser, Michael.

(Update) Watched trailer. Film looks as bad as I remember and image is not being resized/enhanced. It's being 'developed' in some way. Time to let it go.

[Andrius]

I'm not seeing the ML Super Resolution option in the Image menu. I'm on a 2017 13-inch MBP 4TB3. Any ideas?

So the update is still not appearing on the Mac App Store — we've reported this to the App Store team and we hope it'll be fixed ASAP. In the meantime, we've released a beta version of Pixelmator Pro with these features and a few more goodies. If you're on the beta program, you can simply get the update through the app. And if you're not, you can sign up for the beta here.

[Andrius]

The update went live sometime in the evening. Yay! Hope you've all been able to download it now.

[hupey]

I installed the update on my Macbook Pro 13 inch (early 2015) and just like the ML Denoise feature that you introduced a couple of versions earlier, the ML Super Resolution filter does not work on my mac. It takes about 3 minutes for every picture to process and the result is not the same as nothing but very close to nothing. It changes how some pixels look but even with the sample images you provide i cannot get any real sharpening that is close to your examples. The ML Denoise feature also does not do much. Is there any setting i have to change? In your explanations it seemed like all macs that support Metal can do the processing albeit needing more time for it but in the end the effect should still be the same.

[Andrius]

I installed the update on my Macbook Pro 13 inch (early 2015) and just like the ML Denoise feature that you introduced a couple of versions earlier, the ML Super Resolution filter does not work on my mac. It takes about 3 minutes for every picture to process and the result is not the same as nothing but very close to nothing. It changes how some pixels look but even with the sample images you provide i cannot get any real sharpening that is close to your examples. The ML Denoise feature also does not do much. Is there any setting i have to change? In your explanations it seemed like all macs that support Metal can do the processing albeit needing more time for it but in the end the effect should still be the same.

Without seeing examples of the types of images you're upscaling, it's difficult to say what could be going wrong. The examples on our blog are all 100% real but it's also important to note that, at least for now, the algorithm cannot introduce details that aren't already in the image. It's really good at recognizing small details and upscaling them realistically (unlike the classic algorithms) but the data does have to be in the image. Also, the algorithm isn't designed to sharpen blurry images that have already been upscaled using one of the traditional algorithms. In situations like that, it might be best to first downscale the image, then use ML Super Resolution to upscale it.

As for ML Denoise, some examples would also help. The 1.5.4 update includes version 2 of the ML Denoise algorithm (or 1.1, at least) and Pixelmator Pro 1.5.5 will bring ML Denoise v1.2 with some more improvements, especially for very heavy image noise. If you have some images that you could share with us that we could use for testing, I can promise we'd take them into account when making improvements. Feel free to email any sample images to [email protected] and we'll pass them on to the devs.

[OpenBanboo]

Does this feature use CoreML?

[giovanni gallucci]

This looks amazing. I’m a long time user of Pixelmator on MacOS and Pixelmator Pro on iOS and more recently iPadOS. I am in the process of moving off macOS and converting all my computing to iOS and iPadOS. Any plans to bring this technology to iPadOS? Pretty please?

-giovanni

[Dave Kosiur]

Just downloaded the trail version (1.5.5) and am running it on a 5kRetina iMac under High Sierra. When I went to the Image Size window to try your new resizing option, it did not show up. Only the older methods - Lanczos, Nearest Neighbor, etc. -- were there.

Does this new resizing method depend on the newer OS, or some hardware my iMac doesn't have?

Thanks

[Republiken]

Do I understand the super resolution tool?

I have an image. 46,33x30,69 centimetres.
I change the image size to 21cmx14cm.
The resolution is now 118,11.
I choose ML Super resolution and change the resolution from 118,11 to 300.
The image size for some reason changes to 41,99x28cm.
I change it back to the size I want, 21x14.
Is this the way I should work?
If I am impressed?
I don't know yet. Will have to practise.

[st3f]

Does this new resizing method depend on the newer OS, or some hardware my iMac doesn't have?

It's likely to be OS related. The blog post suggests that ML Super Resolution uses Core ML 3 which, I think, came along with macOS Catalina.

[Michael P]

Nice feature
Wouldn't it be nice to have it as an action for batch processing in Automator?

[Michael P]

Just downloaded the trail version (1.5.5) and am running it on a 5kRetina iMac under High Sierra. When I went to the Image Size window to try your new resizing option, it did not show up. Only the older methods - Lanczos, Nearest Neighbor, etc. -- were there.

Does this new resizing method depend on the newer OS, or some hardware my iMac doesn't have?

Thanks

It's an other entry in the Image menu: use ML Super Resolution instead of Resize

[Dave Kosiur]

The ML Super Resolution doesn't show up in my version. Must be because I'm running an older OS, without Core ML, as someone else mentioned.

And I'm reluctant to upgrade to Catalina because I have 32-bit apps that I still depend on, and they've not be upgraded to 64-bit.

Oh well. Some other year ...

[DanThePhotoMan]

Can’t wait to try it out! And just so this is clear to everyone, it is not designed to make your image sharper, but it is supposed to keep the image quality the same when enlarging am image. Photos normally degrade when enlarged. If you don’t need a large photo, there is no need to use this feature. I see it being fantastic for large prints or if quite a bit of cropping needs to be done on a photo.

[Kelvin N.]

Shouldn’t it be 1.1 megaFLOPS?

[ResLes]

Can’t wait to try it out! And just so this is clear to everyone, it is not designed to make your image sharper, but it is supposed to keep the image quality the same when enlarging am image. Photos normally degrade when enlarged. If you don’t need a large photo, there is no need to use this feature. I see it being fantastic for large prints or if quite a bit of cropping needs to be done on a photo.

I thought about the print one, too. I haven’t thought about the crop one. That is really huge. Thanks for mention that.

Can anybody imagine in which other scenarios I could use this feature?

@ Andrius, great to see features which where just recently possible through the OS innovation. Amazing surprise! That is showing that the heart of the app has Mac DNA.

1. Before Pixelmator Pro 2.0 comes out, do you guys address requested features like adjustment layers, multiple colour replace possibilities, new colour picker and releasing the clarity ML and Denoise ML with sliders to work with? Thanks.

[Andrius]

Does this feature use CoreML?

Yes, and it's optimized for Core ML 3 to take advantage of multi-GPU support, making this feature much faster on devices with a dedicated GPU.

This looks amazing. I’m a long time user of Pixelmator on MacOS and Pixelmator Pro on iOS and more recently iPadOS. I am in the process of moving off macOS and converting all my computing to iOS and iPadOS. Any plans to bring this technology to iPadOS? Pretty please?

-giovanni

This feature is very memory-intensive so it may not be possible to bring it to iPadOS due to the differences in memory management, but we'll most certainly try.

Just downloaded the trail version (1.5.5) and am running it on a 5kRetina iMac under High Sierra. When I went to the Image Size window to try your new resizing option, it did not show up. Only the older methods - Lanczos, Nearest Neighbor, etc. -- were there.

Does this new resizing method depend on the newer OS, or some hardware my iMac doesn't have?

Thanks

It requires macOS Mojave or macOS Catalina, so it won't work on High Sierra — apologies for not making that clearer!

Do I understand the super resolution tool?

I have an image. 46,33x30,69 centimetres.
I change the image size to 21cmx14cm.
The resolution is now 118,11.
I choose ML Super resolution and change the resolution from 118,11 to 300.
The image size for some reason changes to 41,99x28cm.
I change it back to the size I want, 21x14.
Is this the way I should work?
If I am impressed?
I don't know yet. Will have to practise.

The print size of the image shouldn't change, only the resolution/pixel dimensions should — when using the ML Super Resolution menu command at least. You could also open up Image > Image Size and enter a custom size/resolution, choosing the ML Super Resolution algorithm for resizing.

Nice feature
Wouldn't it be nice to have it as an action for batch processing in Automator?

It would! We might be able to do this, actually.

Can’t wait to try it out! And just so this is clear to everyone, it is not designed to make your image sharper, but it is supposed to keep the image quality the same when enlarging am image. Photos normally degrade when enlarged. If you don’t need a large photo, there is no need to use this feature. I see it being fantastic for large prints or if quite a bit of cropping needs to be done on a photo.

Yes, that's a very good point! Also, it won't do much to images that have already been enlarged and are blocky or blurry because the algorithm will interpret the blocks or blurred edges as textures. You could try downscaling the image first and then upscaling using ML Super Resolution.

Shouldn’t it be 1.1 megaFLOPS?

You're absolutely right, fixed!

@ Andrius, great to see features which where just recently possible through the OS innovation. Amazing surprise! That is showing that the heart of the app has Mac DNA.

1. Before Pixelmator Pro 2.0 comes out, do you guys address requested features like adjustment layers, multiple colour replace possibilities, new colour picker and releasing the clarity ML and Denoise ML with sliders to work with? Thanks.

I really hate spoiling surprises but, seeing as you're asking:

1. Adjustment layers — maybe in the second half of next year? We need to finish up the color adjustment feature set first and a few foundational changes need to be made to bring the Clarity adjustment as well as improved shadows and highlights.
2. Replacing multiple colors — this is a UI thing as this feature could already work this way from a technological standpoint. And we'd also need to add it to Pixelmator Photo due to the shared codebase. But thanks for the reminder.
3. Color picker — in development at the moment but it doesn't look like it'll come before the new year. It's going to be really cool, though, so it's worth the wait.
4. ML Denoise — the 1.5.4 update brought some improvements to ML Denoise, and 1.5.5 will bring even more. The results should be much more dramatic after these updates (especially in images with heavy noise) but a slider won't appear just yet.

[mr.b33tr00t]

Please add this feature for batch processing in Automator as suggested by Michael P

[Yuka W]

It’s no secret that we’re pretty big fans of machine learning and we love thinking of new and exciting ways to use it in Pixelmator Pro. Our latest ML-powered feature is called ML Super Resolution, released in today’s update, and it makes it possible to increase the resolution of images while keeping them stunningly sharp and detailed. Yes, zooming and enhancing images like they do in all those cheesy police dramas is now a reality!



Read the full blog entry here.

Are you still developers or already data scientists? Good job on ML Super Resolution. From bioinformatician.