Nvidia has been doing some interesting stuff with AI-based upscaling. Supposedly there’s hardware-based AI upscaling in their Shield TV product, and possibly in future game platforms. I haven’t researched it much yet, but if it’s as good as it seems it would be great to see an implementation that works with TouchDesigner.
Anyone who knows more about I’d love to hear from you.
The way it works currently is you need the ‘train’ the model with ultra-high end renders first. So if you want a 4K output, you’d train the model with something like 8K offline, and then you can render real-time at 1080p and get a nice 4K output.
Unfortunately, this process is currently reserved for games (and internally to Nvidia) since it works on a title-by-title basis, it’s not a general solution yet. We’ll see where things go though!
Several months ago I worked with a team on doing a comprehensive AI up-scaling report - comparing a whole host of AI approaches against conventional up-scaling techniques. It’s certainly true that you can get there, but it’s certainly not easy to configure or get up and running - the model training time is also not insignificant, and your results are best when a model is well trained against the content type you’re working with. For example - up-scaling video of waves with a model that’s mostly been trained on trees doesn’t make for great results.
Other interesting artifacts here are that temporal stability is harder to achieve than you might hope. Enlarging a single image is different than enlarging a series of images that play contiguously. It’s easy to end up with flickering or artifacts that break the viewers attention.
There’s for sure some really exciting progress being made here, but having done a deep dive I can say that it definitely takes careful tuning and model training to get results that are decent.
The results I saw weren’t spectacular, but RunwayML has a generic upscaling AI model that I was experimenting with a few weeks ago that can be run through touchdesigner. Theoretically, you can train your own models with their software as well, but I haven’t tried that yet.
Hi,
For a theater project, i need to work on old photos and videos.
I implemented some tools. I begun to make Touchdesigner TOPs, but my version is the free one,
so i made my tools with comand line.
I posted one of this tool ColorizationTop recently to colorize images.
One of the others tool is Super Resolution that permits to upscale resolution. That works quite good for my utilisation.
I began to develop a SuperResTOP but it crashes when i change the scale factor in TD, i think it is a problem with the c++ top outptut. It is working without changing parameters or unloading, changing parameters and loading the plug…
I can share it, but i do not have time to solve the problem now.
SuperRes is a contrib module of opencv, using AI models. I hacked it to use cuda dnn.
Concerning my project i implemented, using opencv dnn cuda, this tools :
All that tools, apart DAIN app, work with comand line.
Compiled with Vs Studio 2017, opencv 4.2 with Cuda dnn.
Ok,
I put in github the tools i implemented to work on a future theater project.
One of the tools, SuperRes.exe upscale resolutions for images and videos.
It uses opencv contrib module dnn_superres and use different models.
You can try it :
Tell me if you try what are the results.