AI upscaling and faster upscaler with Nvidia Maxine SDK

Hello!
I see Touchdesigner support some Nvidia features provided from Maxine SDK, that is great, anyway and unfortunately there is no support yet to 2 great features of that SDK.

  • Super resolution (up to 4X scaling factor using AI)
    The Super Resolution filter, while preserving the content, improves the resolution of low-resolution videos, enhances the details, and sharpens the output.

  • Upscaler
    This is a very fast and light-weight method for upscaling an input video.

It also provides a sharpening parameter to sharpen the resulting output. This feature can be optionally pipelined with the encoder Artifact reduction feature to enhance the scale while reducing the video artifacts. Upscale supports any input resolution and can be upscaled 4/3x, 1.5x, 2x, 3x, or 4x. The output resolution values must be integers, and the ratio of widths must exactly equal the ratio of heights.

Some plan, to add support soon?

2 Likes

+1

Sharpen/Upscale TOPs would be incredible. There are a handful of video exporting apps that do this sort of thing and charge a few hundred dollars per license. I bet these features alone, plus some video exporting components (ala QFH), would bring a fair bit of interest and intrigue to the TouchDesigner platform, especially from the After Effects/Resolve/Premiere video/mograph community.

1 Like

NVIDIA Maxine is real time!! what is a lot better of all what already exist on the market.

I own a Topaz license. It is far from being real-time (aka less than half a frame a second in most cases) and they can charge that much because of how they train models / what models are made available :sweat_smile:

Now, what we noticed with Maxine is that it is mostly trained against the content of one person sitting in front of a webcam, for video conferencing / Twitch… etc

We’d have to test probably how the SR model works but it could be that the results would introduce artifacts rather than upscaling and couldn’t cover the variety of content produced in TouchDesigner. Just like a high-end app like Topaz does introduce more artifacts sometimes rather than upscaling/enhancing the frames…

Now there are a few other solutions. NVidia IS, AMD FSR, and Nvidia DLSS.

You could give a try to Nvidia IS since it is a Driver based solution (you’ll find it in the latest drivers and in Experience).

They also provide an IS SDK, just like AMD provides FSR SDK… but those need to be built into the app and again the main issue we could expect is that it doesn’t work well with TouchDesigner content, but tbd…

Lastly, there is DLSS which is much more involved in the whole pipeline as far as I know.

There is a good summary here with comparisons between the enhancers Nvidia Image Scaling vs AMD FSR vs DLSS - Which Works Best in Games? - YouTube

1 Like

Thanks for the suggestion. I’ve experimented with the SuperRes filter a little when we implemented the Background/Denoise TOPs and it is something we’re considering, but we don’t have any immediate plans for implementation at this point.

One obstacle is just the sheer size of the ai models - the superres models would add about 300Mb to our installer just for that one feature. And, as Michel says, we’ve got to balance that against other potential solutions that could work better and in more cases.

All of the Maxine stuff is still technically in beta, so we’re keeping an eye on it and it’s always good to hear what you guys are interested in as we plan things going forward.

2 Likes

NVIDIA Maxine SR use convolutional autoencoder network and can be used with any kind of content as input based on my tests.
Are you sure is only trained with persons in front of a cam?
When I test this some time ago, I have great results with any kind of input content.
FSR is really ugly for standard video, DLSS not apply to standard videos due the nature of how works and what do etc, all other sharpen libs form Nvidia produce poor results compared with Maxine SR.

Great to know is under your eyes :slight_smile:
I have under me eyes buy licennse of TD asap!!! :slight_smile:

Note: 300MB at this days for sure is nothing :grinning:
Anyway is think can be a problem…then can be optional package?

Ableton without all libs is little, then you can install extra addons, the limit is what you want download based on your needs and size of your storage lol.
Best regards!!

300Mb doesn’t sound like a lot these days, but it does all add up and puts barriers to people being able to download and try out the software easily. I’m told there are also some compatibility limits with the installer once we go much over 2Gb that create additional issues.

Supporting optional download components is high on our list at the moment since a lot of the AI-based solutions (Kinect, FaceTrack, Maxine, etc) take up a lot of space but only get used by a small percentage of our users.

3 Likes

Optional download components panel would be amazing. +1 to that!

4 Likes

Thanks for your time to explain, i understand what you said.
Well, hope to see Maxine Super Resolution and fast upscaler soon.
Dont forgot nvidia body tracker too : )

One of the problem we are having with deciding which way to go with that is Nvidia does not allow us to deliver their models as a separate discrete download, due to licensing. We are discussing the possibilities though as the Windows installer is becoming an untenable situation.

1 Like

I understand you guys want have a installer compact a reduced in size, anyway I insist big installers not are a real problem with modern connectivity, just with my 4G can download few gigs in few minutes. *in case i stay away of my desktop)
DAWS, 3D Modeling Software, CAD Software, 3D ENGINES etc, are giant and of course would be nice just use few MB instead GB
But well is my opinion only, I respect and understand what other users think.

Anyway, the benefit of process all and render in real time at 1080p or 1440p and then upscale the out to pseudo 4k with less penalty overall process, can be a big advantage for very heavy projects.
Of course is pseudo 4k, Maxine is a good on steroids sharpen. No is the panacea but is really good.

I think a feature like this will be used for more people of what we imagine now.
From my point of view the best goal of this is something similar like in gaming world with DLSS, where you gain in performance and in case of TouchD, that performance can be free resources for a complex projects.

I new on touch designer, I start build with free version a replacement of OBS studio, the reason is TD allow to me as a coder/creator to do some very special things are impossible on OBS.
I post in some moment, some info of what I doing with TD here in the forum, and on my site forum.
savethehuman5 . com (when I be able to publish links here in the forum?)
I plan redirect my software on video direction this 2022, adding some features make of my chameleon software a more video friendly tool.
Anyway when I made public in some months my TouchDesigner project I think more people will be atracted to learn TD with live streaming purposes. Hoppe for that day count with Maxine SR and my lic for TD :slight_smile:
Love this software!!! dont know why I not use before!

1 Like

Thanks for the kind words @SaveTheHuman5! And of course your feedback.
Regarding sharing your work, we’d recommend now using the COMMUNITY blogs/gallery and different post types available to you there. When you login to MY ACCOUNT you will see all the options for creating your own posts. Also be sure to fill out your Profile Page so people can find out about you and all your articles in one place. You will see this under MY ACCOUNT > MY PUBLIC PROFILE.
Welcome to the community!

Would love a DLSS top, is that within the realm of attainability?

Done!! thk!

@drmbt DLSS is something we’ve discussed, but we don’t have any immediate plans for it. From my understanding, it would require some significant changes to our render pipeline to generate the inputs that the system needs and for the moment we’re focused on the Vulkan transition.

@SaveTheHuman5 the nvidia body tracking library wasn’t available in the version of the AR sdk that we’re currently using, but we are keeping an eye on it as well. I know people have used it inside touch already via a notch plugin.

+1 on the body tracking. Game changer!

Sorry to hijack this old thread, but I’d also be very interested in an implementation of the Maxine body tracker in Touch!

Thanks for the feedback. It is on our near-term planning list to take another look at maxine body tracking. We’ll let you guys know how it goes.

1 Like