Vulkan Device Error when running project 24/7

Bugs
1

Left my project runing for some time (not sure how long exactly but lets say about a day) but when coming back I got the following crash-message:

image_6508

The only thing the project does is:

  • receive a 24/7 live-stream
  • de-embed the audio
  • route it to/frome an audio-processing software over virtual audio cable
  • meanwhile, buffer the video in a cacheTOP so it gets delayed the same amount of time as the audio
  • re-embed the processed audio with the video so they are lipsync

The previous version i was running on (2021.13610) did not seem to have this issue running for long periods.

  1. Does anyone had the same/similar problems?
  2. Are there any recommendations/do’s&don’ts on running a touchdesigner project for long periods? (as the plan is to let this project run for 24/7…)
2

It could be a sign of a memory leak.

Are you using any custom GLSL in TOP / MATs ?

What you could do to possibly find the leak is remove part of your network while testing: maybe try with the cacheTOP first (what’s the current cache size ?), how is the live stream received ? Video Stream In ? What parameters?

If you look at the GPU memory used by the process (details view of Task Manager, add the GPU memory column), do you see it continuously increasing over time ? After 10 minutes… 1 hour… 4 hours ?

Are there any recommendations/do’s&don’ts on running a touchdesigner project for long periods? (as the plan is to let this project run for 24/7…)

Usually on production projects running 24/7, some developers would schedule a reboot of the machine at some point, or restart of the TouchDesigner process at a time X, using various process managers, such as NodeJS’ PM2.

That being said, it might “hide the leak” in a way where the leak would still be happening but not fill all the RAM or GPU Memory before reboot / restart of process… It doesn’t mean that the leak is not here and that it shouldn’t be fixed.

Best,
Michel