All about the new ML Super Resolution feature in Pixelmator Pro

Discuss the latest Pixelmator news.
User avatar

2019-12-19 15:21:18

by mr.b33tr00t 2019-12-19 11:18:58 Please add this feature for batch processing in Automator as suggested by Michael P
Noted!
by Yuka W 2019-12-19 11:59:55 Are you still developers or already data scientists? Good job on ML Super Resolution. From bioinformatician.
Why not both?
User avatar

2019-12-19 15:25:23

by Andrius 2019-12-19 15:22:53
Noted!



Why not both?
Scientists usually publish papers. Have you?
User avatar

2019-12-19 15:27:50

by Yuka W 2019-12-18 21:00:00 Scientists usually publish papers. Have you?
OK OK, we're just app developers...
User avatar

2019-12-19 15:33:18

by Andrius 2019-12-19 00:00:00
OK OK, we're just app developers...
You may consider collaborating with professional scientists in ML field. I know medical biologists working with image analysis. Communicating with them may result in scientific publications which will add credibility to your apps and may be useful to them too.
User avatar

2019-12-19 21:07:48

I wonder if you trained it on specific kinds of images -- such as people or faces -- whether it could do a better job of filling in missing details on parts of images that it recognizes. Or fixing blemishes.

Also, have you considered using this to upscale old videos to better resolution? Not a feature for Pixelmator, unless you just want to extract a better-resolution image of a single frame (maybe also using some inter-frame info?). But perhaps a cool new product.
User avatar

2019-12-19 22:55:14

This is amazing! I am curious if you also incorporated wavelet-based reconstruction techniques along with the machine learning?

And I wanted to bring your attention to "Let's Enhance": https://www.youtube.com/watch?v=Vxq9yj2pVWk
User avatar

2019-12-20 08:59:22

by Andrius
Yes, and it's optimized for Core ML 3 to take advantage of multi-GPU support, making this feature much faster on devices with a dedicated GPU.



This feature is very memory-intensive so it may not be possible to bring it to iPadOS due to the differences in memory management, but we'll most certainly try.



It requires macOS Mojave or macOS Catalina, so it won't work on High Sierra — apologies for not making that clearer!



The print size of the image shouldn't change, only the resolution/pixel dimensions should — when using the ML Super Resolution menu command at least. You could also open up Image > Image Size and enter a custom size/resolution, choosing the ML Super Resolution algorithm for resizing.



It would! We might be able to do this, actually.



Yes, that's a very good point! Also, it won't do much to images that have already been enlarged and are blocky or blurry because the algorithm will interpret the blocks or blurred edges as textures. You could try downscaling the image first and then upscaling using ML Super Resolution.



You're absolutely right, fixed!



I really hate spoiling surprises but, seeing as you're asking:

1. Adjustment layers — maybe in the second half of next year? We need to finish up the color adjustment feature set first and a few foundational changes need to be made to bring the Clarity adjustment as well as improved shadows and highlights.
2. Replacing multiple colors — this is a UI thing as this feature could already work this way from a technological standpoint. And we'd also need to add it to Pixelmator Photo due to the shared codebase. But thanks for the reminder.
3. Color picker — in development at the moment but it doesn't look like it'll come before the new year. It's going to be really cool, though, so it's worth the wait.
4. ML Denoise — the 1.5.4 update brought some improvements to ML Denoise, and 1.5.5 will bring even more. The results should be much more dramatic after these updates (especially in images with heavy noise) but a slider won't appear just yet.

Thanks Andrius for the honest and detailed update. Know I have a kind of an idea when to expect that.

By the way you got really good press with the new super resolution feature.

Well done
User avatar

2019-12-20 10:04:57

by Norm Margolus 2019-12-19 21:07:48 I wonder if you trained it on specific kinds of images -- such as people or faces -- whether it could do a better job of filling in missing details on parts of images that it recognizes. Or fixing blemishes.
If the goal of the algorithm was to synthesize missing details, we'd have to use a completely different approach — rather than a convolutional neural network, it would have to be a generative adversarial network. The problem with that is that it would synthesize details from noise and would generally be much more unpredictable, so it wouldn't have as much of a functional use. We decided to go for predictability and consistency.
Also, have you considered using this to upscale old videos to better resolution? Not a feature for Pixelmator, unless you just want to extract a better-resolution image of a single frame (maybe also using some inter-frame info?). But perhaps a cool new product.
This would also require a different approach to the one we've chosen, specifically to take multiple frames into account to keep the colors/brightness consistent between frames. But it would definitely be cool!
by Evan M 2019-12-19 22:55:14 This is amazing! I am curious if you also incorporated wavelet-based reconstruction techniques along with the machine learning?

And I wanted to bring your attention to "Let's Enhance": https://www.youtube.com/watch?v=Vxq9yj2pVWk
No wavelet-based reconstruction techniques, most of what we did is mentioned in the blog. That Let's Enhance video is awesome, by the way!
by ResLes 2019-12-20 06:11:28 Thanks Andrius for the honest and detailed update. Know I have a kind of an idea when to expect that.

By the way you got really good press with the new super resolution feature.

Well done
Thanks!
User avatar

2019-12-22 09:46:28

I have seen that ml resolution is also available in the image resize level. Does it downscale with a bigger file size then the other algorithms to keep the quality high or is the size the same but still better quality? In other words world you use it as default to resize an image? Of course, I could test it but if it does, would like to know when I should use it. Thanks
User avatar

2019-12-24 05:16:05

I wonder why do you call it "super resolution"? Why not "high" or, let's say, "hyper"? What's in the name?
User avatar

2019-12-24 09:03:57

I have been experimenting with ML Super Resolution now and I am getting used to it. I must say I like it a lot and consider it to be fantastic! I must say congratulations once again fo what you are doing to Pixelmator Pro! I look forward to the coming decade!
User avatar

2019-12-27 11:14:24

by ResLes 2019-12-22 09:46:28 I have seen that ml resolution is also available in the image resize level. Does it downscale with a bigger file size then the other algorithms to keep the quality high or is the size the same but still better quality? In other words world you use it as default to resize an image? Of course, I could test it but if it does, would like to know when I should use it. Thanks
ML Super Resolution is designed for upscaling images only — for downsampling, you're best off using one of the other algorithms.
by Yuka W 2019-12-24 05:16:05 I wonder why do you call it "super resolution"? Why not "high" or, let's say, "hyper"? What's in the name?
Super-resolution is actually just the standard term used for any action that creates a higher-resolution image from a low-res one. This can also be done by merging multiple pixel-shifted exposures, for example, so you might also see the term there.
by Republiken 2019-12-24 09:03:57 I have been experimenting with ML Super Resolution now and I am getting used to it. I must say I like it a lot and consider it to be fantastic! I must say congratulations once again fo what you are doing to Pixelmator Pro! I look forward to the coming decade!
We're looking forward to the coming years too! And we're especially looking forward to the improvements we have lined up for the next few months.
User avatar

2019-12-27 11:17:42

by Andrius 2019-12-27 11:15:31
ML Super Resolution is designed for upscaling images only — for downsampling, you're best off using one of the other algorithms.



Super-resolution is actually just the standard term used for any action that creates a higher-resolution image from a low-res one. This can also be done by merging multiple pixel-shifted exposures, for example, so you might also see the term there.



We're looking forward to the coming years too! And we're especially looking forward to the improvements we have lined up for the next few months.
I am so excited to see you guys are not slowing down and your flagship product is going to be better and better...
User avatar

2020-01-02 14:41:17

Happy New Year pixelmator folks... wondering if this feature will benefit with an nvidia eGPU as it does with the and eGPU you tested it with.
User avatar

2020-01-02 15:12:44

Happy New Year to too! Officially, only AMD eGPUs are supported in macOS as outlined on the following page:

https://support.apple.com/HT208544

So you'd need to use one of those GPUs, at least for now.