Nvidia Upscaler error - Failed to initialize CUDA - Failed to set CUDA context

Hello,

I got this error and no idea why (2023.11880). Does this also happen to anyone else? We use 2 x gpus :1 rtx 4090 + 1 rtx 3060, both are connected to separate displays. This does not seem to be the issue though. Sometime I create a new file and add the upscaler TOP and it’s fine. Then sometime not. Then if I save a toe with the upscaler and reopen, I got the error again. I even copied pasted my whole project to a new file and the upscaler was working. But then I saved and reopen and it does not work. I work on the same monitor and all, does not seem to make a difference if I switch anyway. I have CUDA installed on windows btw. If anyone have ideas / suggestions! I’m a bit puzzled by this error tbh.

Thanks,

G.

I suspect this is still an issue of CUDA trying to initialize on the wrong device. Have you tried using the GPU affinity system to assign TouchDesigner to a specific gpu? Using Multiple Graphic Cards - Derivative

Hey Rob,

Thanks for the quick reply! Yes I’ve tried to start TD with the -gpuformonitor flag but it also fails, the same way. I’m not 100% sure to understand the GPU affinity part tbh.

This is what the monitor DAT shows:


Is affinity working with RTX (non “quadro”) cards? Is there a way to force it? I’ve tried to launch the toe with gpuformonitor 1, but same result basically. For info the monitor plugged on the 3060 is the main display in windows.

Thx

Affinity should work with the RTX cards in that build. Do you see any difference if you launch it with gpuformonitor 0 and try to stick with the 4090 card?

It might help to narrow things down if we can confirm whether CUDA is failing anywhere else or if it is just Face Tracking. If you try to save a movie with the HapHDR codec in the Movie File Out that should also trigger the same CUDA initialization. It would be particularly useful to know if that also fails after the face tracking has already failed.

Thanks for the advice Rob. So I’ve tried what you suggested, and if I start just the .toe with gpuformonitor 0 or gpuformonitor 1 it works just fine, no error with the nvidia upscale TOP. BUT if I start both files at the same time (we got 2 different .toe that should run at the same time but on separated GPU), one with the flag gpuformonitor 0 and the other one (the one with the nivida upscale TOP) with gpuformonitor 1 then I got the error again. I I close both, start the file with the upscale TOP again (with or without flag) then it works again…

Starting both toe with gpuformonitor 0 works, but everything is on the 4090 of course…

Also tried the HapHDR codec in the Movie File Out (when I got the error with the upscale TOP) and this is the error I have then :arrow_down:

If I close everything, start a new TD instance, then add the movie file out and save a HapHDR, no problem at all. Without any flags.

And, last but not least, I close everything, start the .toe with the upscale TOP (affinity is true for the 3060) then the errror pops up again. I keep it open, start a new TD instance, add the upscale TOP, no error (affinity is also true for the 3060) :scream_cat:

Thanks for the extra information. I’ve asked around the team here and it feels like there might potentially be a conflict between how CUDA and Vulkan select between the different GPUs.

Would you be able to run ‘vulkaninfo.exe’ and send us the results? It is usually installed automatically to C:\windows\system32