All about the new ML Super Resolution feature in Pixelmator Pro

[Andrius]

It’s no secret that we’re pretty big fans of machine learning and we love thinking of new and exciting ways to use it in Pixelmator Pro. Our latest ML-powered feature is called ML Super Resolution, released in today’s update, and it makes it possible to increase the resolution of images while keeping them stunningly sharp and detailed. Yes, zooming and enhancing images like they do in all those cheesy police dramas is now a reality!







Read the full blog entry here.

[st3f]

like they do in all those cheesy police dramas

I think that Blade Runner beat them to it by a decade or two. There's also a film where the protagonist's face is on an unclear photograph from a murder scene. The image is being 'enhanced'. He only has a few hours of freedom to clear his name before the program completes and makes him the prime suspect. I wish I could remember the film.
I'm looking forward to using this... but for more mundane purposes, though.

[Andrius]

By the way, it looks like the update is taking a lot longer than usual to appear in the Mac App Store — but it's coming. In the meantime, I'll leave this little gem here for your viewing pleasure.

[ChrisRosa]

I'm not seeing the ML Super Resolution option in the Image menu. I'm on a 2017 13-inch MBP 4TB3. Any ideas?

[gvisser]

Thanks Pixelmator!
This is a feature I've been wanting for years!

st3f The movie you are thinking of is "No Way Out" with Kevin Costner.

[Michael Long]

I think that Blade Runner beat them to it by a decade or two. There's also a film where the protagonist's face is on an unclear photograph from a murder scene. The image is being 'enhanced'. He only has a few hours of freedom to clear his name before the program completes and makes him the prime suspect. I wish I could remember the film.
I'm looking forward to using this... but for more mundane purposes, though.

No Way Out. Kevin Costner, Sean Young.

[st3f]

st3f The movie you are thinking of is "No Way Out" with Kevin Costner.

No Way Out. Kevin Costner, Sean Young.

Now... the question is... should I
Option A
1. Track down the film.
2. Work out when it's set and how long the rendering takes in film.
3. Estimate the pixel dimension and the quality of the rendering hardware used.
4. Work out how long it takes today on current equivalent hardware.
5. Plot some Moore's law graph with two data points and a lot of supposition.

Option B
Let sleeping dogs lie as I seem to recall the film was a bit of a stinker.

I've found it to rent on Amazon Prime for £3.49. Is it worth £3.49 and 1h54m of my life to pursue Option A? Hmmm... I'm going to watch the trailer and make my mind up after that.

Thanks for the responses, gvisser, Michael.

(Update) Watched trailer. Film looks as bad as I remember and image is not being resized/enhanced. It's being 'developed' in some way. Time to let it go.

[Andrius]

I'm not seeing the ML Super Resolution option in the Image menu. I'm on a 2017 13-inch MBP 4TB3. Any ideas?

So the update is still not appearing on the Mac App Store — we've reported this to the App Store team and we hope it'll be fixed ASAP. In the meantime, we've released a beta version of Pixelmator Pro with these features and a few more goodies. If you're on the beta program, you can simply get the update through the app. And if you're not, you can sign up for the beta here.

[Andrius]

The update went live sometime in the evening. Yay! Hope you've all been able to download it now.

[hupey]

I installed the update on my Macbook Pro 13 inch (early 2015) and just like the ML Denoise feature that you introduced a couple of versions earlier, the ML Super Resolution filter does not work on my mac. It takes about 3 minutes for every picture to process and the result is not the same as nothing but very close to nothing. It changes how some pixels look but even with the sample images you provide i cannot get any real sharpening that is close to your examples. The ML Denoise feature also does not do much. Is there any setting i have to change? In your explanations it seemed like all macs that support Metal can do the processing albeit needing more time for it but in the end the effect should still be the same.

[Andrius]

I installed the update on my Macbook Pro 13 inch (early 2015) and just like the ML Denoise feature that you introduced a couple of versions earlier, the ML Super Resolution filter does not work on my mac. It takes about 3 minutes for every picture to process and the result is not the same as nothing but very close to nothing. It changes how some pixels look but even with the sample images you provide i cannot get any real sharpening that is close to your examples. The ML Denoise feature also does not do much. Is there any setting i have to change? In your explanations it seemed like all macs that support Metal can do the processing albeit needing more time for it but in the end the effect should still be the same.

Without seeing examples of the types of images you're upscaling, it's difficult to say what could be going wrong. The examples on our blog are all 100% real but it's also important to note that, at least for now, the algorithm cannot introduce details that aren't already in the image. It's really good at recognizing small details and upscaling them realistically (unlike the classic algorithms) but the data does have to be in the image. Also, the algorithm isn't designed to sharpen blurry images that have already been upscaled using one of the traditional algorithms. In situations like that, it might be best to first downscale the image, then use ML Super Resolution to upscale it.

As for ML Denoise, some examples would also help. The 1.5.4 update includes version 2 of the ML Denoise algorithm (or 1.1, at least) and Pixelmator Pro 1.5.5 will bring ML Denoise v1.2 with some more improvements, especially for very heavy image noise. If you have some images that you could share with us that we could use for testing, I can promise we'd take them into account when making improvements. Feel free to email any sample images to support@pixelmator.com and we'll pass them on to the devs.

[OpenBanboo]

Does this feature use CoreML?

[giovanni gallucci]

This looks amazing. I’m a long time user of Pixelmator on MacOS and Pixelmator Pro on iOS and more recently iPadOS. I am in the process of moving off macOS and converting all my computing to iOS and iPadOS. Any plans to bring this technology to iPadOS? Pretty please?

-giovanni

[Dave Kosiur]

Just downloaded the trail version (1.5.5) and am running it on a 5kRetina iMac under High Sierra. When I went to the Image Size window to try your new resizing option, it did not show up. Only the older methods - Lanczos, Nearest Neighbor, etc. -- were there.

Does this new resizing method depend on the newer OS, or some hardware my iMac doesn't have?

Thanks

[Republiken]

Do I understand the super resolution tool?

I have an image. 46,33x30,69 centimetres.
I change the image size to 21cmx14cm.
The resolution is now 118,11.
I choose ML Super resolution and change the resolution from 118,11 to 300.
The image size for some reason changes to 41,99x28cm.
I change it back to the size I want, 21x14.
Is this the way I should work?
If I am impressed?
I don't know yet. Will have to practise.

[st3f]

Does this new resizing method depend on the newer OS, or some hardware my iMac doesn't have?

It's likely to be OS related. The blog post suggests that ML Super Resolution uses Core ML 3 which, I think, came along with macOS Catalina.

[Michael P]

Nice feature
Wouldn't it be nice to have it as an action for batch processing in Automator?

[Michael P]

Just downloaded the trail version (1.5.5) and am running it on a 5kRetina iMac under High Sierra. When I went to the Image Size window to try your new resizing option, it did not show up. Only the older methods - Lanczos, Nearest Neighbor, etc. -- were there.

Does this new resizing method depend on the newer OS, or some hardware my iMac doesn't have?

Thanks

It's an other entry in the Image menu: use ML Super Resolution instead of Resize

[Dave Kosiur]

The ML Super Resolution doesn't show up in my version. Must be because I'm running an older OS, without Core ML, as someone else mentioned.

And I'm reluctant to upgrade to Catalina because I have 32-bit apps that I still depend on, and they've not be upgraded to 64-bit.

Oh well. Some other year ...

[DanThePhotoMan]

Can’t wait to try it out! And just so this is clear to everyone, it is not designed to make your image sharper, but it is supposed to keep the image quality the same when enlarging am image. Photos normally degrade when enlarged. If you don’t need a large photo, there is no need to use this feature. I see it being fantastic for large prints or if quite a bit of cropping needs to be done on a photo.

[Kelvin N.]

Shouldn’t it be 1.1 megaFLOPS?

[ResLes]

Can’t wait to try it out! And just so this is clear to everyone, it is not designed to make your image sharper, but it is supposed to keep the image quality the same when enlarging am image. Photos normally degrade when enlarged. If you don’t need a large photo, there is no need to use this feature. I see it being fantastic for large prints or if quite a bit of cropping needs to be done on a photo.

I thought about the print one, too. I haven’t thought about the crop one. That is really huge. Thanks for mention that.

Can anybody imagine in which other scenarios I could use this feature?

@ Andrius, great to see features which where just recently possible through the OS innovation. Amazing surprise! That is showing that the heart of the app has Mac DNA.

1. Before Pixelmator Pro 2.0 comes out, do you guys address requested features like adjustment layers, multiple colour replace possibilities, new colour picker and releasing the clarity ML and Denoise ML with sliders to work with? Thanks.

[Andrius]

Does this feature use CoreML?

Yes, and it's optimized for Core ML 3 to take advantage of multi-GPU support, making this feature much faster on devices with a dedicated GPU.

This looks amazing. I’m a long time user of Pixelmator on MacOS and Pixelmator Pro on iOS and more recently iPadOS. I am in the process of moving off macOS and converting all my computing to iOS and iPadOS. Any plans to bring this technology to iPadOS? Pretty please?

-giovanni

This feature is very memory-intensive so it may not be possible to bring it to iPadOS due to the differences in memory management, but we'll most certainly try.

Just downloaded the trail version (1.5.5) and am running it on a 5kRetina iMac under High Sierra. When I went to the Image Size window to try your new resizing option, it did not show up. Only the older methods - Lanczos, Nearest Neighbor, etc. -- were there.

Does this new resizing method depend on the newer OS, or some hardware my iMac doesn't have?

Thanks

It requires macOS Mojave or macOS Catalina, so it won't work on High Sierra — apologies for not making that clearer!

Do I understand the super resolution tool?

I have an image. 46,33x30,69 centimetres.
I change the image size to 21cmx14cm.
The resolution is now 118,11.
I choose ML Super resolution and change the resolution from 118,11 to 300.
The image size for some reason changes to 41,99x28cm.
I change it back to the size I want, 21x14.
Is this the way I should work?
If I am impressed?
I don't know yet. Will have to practise.

The print size of the image shouldn't change, only the resolution/pixel dimensions should — when using the ML Super Resolution menu command at least. You could also open up Image > Image Size and enter a custom size/resolution, choosing the ML Super Resolution algorithm for resizing.

Nice feature
Wouldn't it be nice to have it as an action for batch processing in Automator?

It would! We might be able to do this, actually.

Can’t wait to try it out! And just so this is clear to everyone, it is not designed to make your image sharper, but it is supposed to keep the image quality the same when enlarging am image. Photos normally degrade when enlarged. If you don’t need a large photo, there is no need to use this feature. I see it being fantastic for large prints or if quite a bit of cropping needs to be done on a photo.

Yes, that's a very good point! Also, it won't do much to images that have already been enlarged and are blocky or blurry because the algorithm will interpret the blocks or blurred edges as textures. You could try downscaling the image first and then upscaling using ML Super Resolution.

Shouldn’t it be 1.1 megaFLOPS?

You're absolutely right, fixed!

@ Andrius, great to see features which where just recently possible through the OS innovation. Amazing surprise! That is showing that the heart of the app has Mac DNA.

1. Before Pixelmator Pro 2.0 comes out, do you guys address requested features like adjustment layers, multiple colour replace possibilities, new colour picker and releasing the clarity ML and Denoise ML with sliders to work with? Thanks.

I really hate spoiling surprises but, seeing as you're asking:

1. Adjustment layers — maybe in the second half of next year? We need to finish up the color adjustment feature set first and a few foundational changes need to be made to bring the Clarity adjustment as well as improved shadows and highlights.
2. Replacing multiple colors — this is a UI thing as this feature could already work this way from a technological standpoint. And we'd also need to add it to Pixelmator Photo due to the shared codebase. But thanks for the reminder.
3. Color picker — in development at the moment but it doesn't look like it'll come before the new year. It's going to be really cool, though, so it's worth the wait.
4. ML Denoise — the 1.5.4 update brought some improvements to ML Denoise, and 1.5.5 will bring even more. The results should be much more dramatic after these updates (especially in images with heavy noise) but a slider won't appear just yet.

[mr.b33tr00t]

Please add this feature for batch processing in Automator as suggested by Michael P

[Yuka W]

It’s no secret that we’re pretty big fans of machine learning and we love thinking of new and exciting ways to use it in Pixelmator Pro. Our latest ML-powered feature is called ML Super Resolution, released in today’s update, and it makes it possible to increase the resolution of images while keeping them stunningly sharp and detailed. Yes, zooming and enhancing images like they do in all those cheesy police dramas is now a reality!







Read the full blog entry here.

Are you still developers or already data scientists? Good job on ML Super Resolution. From bioinformatician.

[Andrius]

Please add this feature for batch processing in Automator as suggested by Michael P

Noted!

Are you still developers or already data scientists? Good job on ML Super Resolution. From bioinformatician.

Why not both?

[Yuka W]

Noted!



Why not both?

Scientists usually publish papers. Have you?

[Andrius]

Scientists usually publish papers. Have you?

OK OK, we're just app developers...

[Yuka W]

OK OK, we're just app developers...

You may consider collaborating with professional scientists in ML field. I know medical biologists working with image analysis. Communicating with them may result in scientific publications which will add credibility to your apps and may be useful to them too.

[Norm Margolus]

I wonder if you trained it on specific kinds of images -- such as people or faces -- whether it could do a better job of filling in missing details on parts of images that it recognizes. Or fixing blemishes.

Also, have you considered using this to upscale old videos to better resolution? Not a feature for Pixelmator, unless you just want to extract a better-resolution image of a single frame (maybe also using some inter-frame info?). But perhaps a cool new product.

[Evan M]

This is amazing! I am curious if you also incorporated wavelet-based reconstruction techniques along with the machine learning?

And I wanted to bring your attention to "Let's Enhance": https://www.youtube.com/watch?v=Vxq9yj2pVWk

[ResLes]

Yes, and it's optimized for Core ML 3 to take advantage of multi-GPU support, making this feature much faster on devices with a dedicated GPU.



This feature is very memory-intensive so it may not be possible to bring it to iPadOS due to the differences in memory management, but we'll most certainly try.



It requires macOS Mojave or macOS Catalina, so it won't work on High Sierra — apologies for not making that clearer!



The print size of the image shouldn't change, only the resolution/pixel dimensions should — when using the ML Super Resolution menu command at least. You could also open up Image > Image Size and enter a custom size/resolution, choosing the ML Super Resolution algorithm for resizing.



It would! We might be able to do this, actually.



Yes, that's a very good point! Also, it won't do much to images that have already been enlarged and are blocky or blurry because the algorithm will interpret the blocks or blurred edges as textures. You could try downscaling the image first and then upscaling using ML Super Resolution.



You're absolutely right, fixed!



I really hate spoiling surprises but, seeing as you're asking:

1. Adjustment layers — maybe in the second half of next year? We need to finish up the color adjustment feature set first and a few foundational changes need to be made to bring the Clarity adjustment as well as improved shadows and highlights.
2. Replacing multiple colors — this is a UI thing as this feature could already work this way from a technological standpoint. And we'd also need to add it to Pixelmator Photo due to the shared codebase. But thanks for the reminder.
3. Color picker — in development at the moment but it doesn't look like it'll come before the new year. It's going to be really cool, though, so it's worth the wait.
4. ML Denoise — the 1.5.4 update brought some improvements to ML Denoise, and 1.5.5 will bring even more. The results should be much more dramatic after these updates (especially in images with heavy noise) but a slider won't appear just yet.

Thanks Andrius for the honest and detailed update. Know I have a kind of an idea when to expect that.

By the way you got really good press with the new super resolution feature.

Well done

[Andrius]

I wonder if you trained it on specific kinds of images -- such as people or faces -- whether it could do a better job of filling in missing details on parts of images that it recognizes. Or fixing blemishes.

If the goal of the algorithm was to synthesize missing details, we'd have to use a completely different approach — rather than a convolutional neural network, it would have to be a generative adversarial network. The problem with that is that it would synthesize details from noise and would generally be much more unpredictable, so it wouldn't have as much of a functional use. We decided to go for predictability and consistency.

Also, have you considered using this to upscale old videos to better resolution? Not a feature for Pixelmator, unless you just want to extract a better-resolution image of a single frame (maybe also using some inter-frame info?). But perhaps a cool new product.

This would also require a different approach to the one we've chosen, specifically to take multiple frames into account to keep the colors/brightness consistent between frames. But it would definitely be cool!

This is amazing! I am curious if you also incorporated wavelet-based reconstruction techniques along with the machine learning?

And I wanted to bring your attention to "Let's Enhance": https://www.youtube.com/watch?v=Vxq9yj2pVWk

No wavelet-based reconstruction techniques, most of what we did is mentioned in the blog. That Let's Enhance video is awesome, by the way!

Thanks Andrius for the honest and detailed update. Know I have a kind of an idea when to expect that.

By the way you got really good press with the new super resolution feature.

Well done

Thanks!

[ResLes]

I have seen that ml resolution is also available in the image resize level. Does it downscale with a bigger file size then the other algorithms to keep the quality high or is the size the same but still better quality? In other words world you use it as default to resize an image? Of course, I could test it but if it does, would like to know when I should use it. Thanks

[Yuka W]

I wonder why do you call it "super resolution"? Why not "high" or, let's say, "hyper"? What's in the name?

[Republiken]

I have been experimenting with ML Super Resolution now and I am getting used to it. I must say I like it a lot and consider it to be fantastic! I must say congratulations once again fo what you are doing to Pixelmator Pro! I look forward to the coming decade!

[Andrius]

I have seen that ml resolution is also available in the image resize level. Does it downscale with a bigger file size then the other algorithms to keep the quality high or is the size the same but still better quality? In other words world you use it as default to resize an image? Of course, I could test it but if it does, would like to know when I should use it. Thanks

ML Super Resolution is designed for upscaling images only — for downsampling, you're best off using one of the other algorithms.

I wonder why do you call it "super resolution"? Why not "high" or, let's say, "hyper"? What's in the name?

Super-resolution is actually just the standard term used for any action that creates a higher-resolution image from a low-res one. This can also be done by merging multiple pixel-shifted exposures, for example, so you might also see the term there.

I have been experimenting with ML Super Resolution now and I am getting used to it. I must say I like it a lot and consider it to be fantastic! I must say congratulations once again fo what you are doing to Pixelmator Pro! I look forward to the coming decade!

We're looking forward to the coming years too! And we're especially looking forward to the improvements we have lined up for the next few months.

[ResLes]

ML Super Resolution is designed for upscaling images only — for downsampling, you're best off using one of the other algorithms.



Super-resolution is actually just the standard term used for any action that creates a higher-resolution image from a low-res one. This can also be done by merging multiple pixel-shifted exposures, for example, so you might also see the term there.



We're looking forward to the coming years too! And we're especially looking forward to the improvements we have lined up for the next few months.

I am so excited to see you guys are not slowing down and your flagship product is going to be better and better...

[andbobisyouruncle]

Happy New Year pixelmator folks... wondering if this feature will benefit with an nvidia eGPU as it does with the and eGPU you tested it with.

[Andrius]

Happy New Year to too! Officially, only AMD eGPUs are supported in macOS as outlined on the following page:

https://support.apple.com/HT208544

So you'd need to use one of those GPUs, at least for now.