We have a project running on a remote site where TD silently crashes with no logs or any trace as to why. We’ve tested the same project here in the studio with the identical setup and the same hardware connected, and we don’t get this result, so it seems to be something local to the environment. We have 3 Orbbecs Femto Bolts running and 2 x 1200 x 1080 projectors for the video output connected, using a RTX 4090 card.
My question is really more general in looking for some guidance as to how we can trace and understand what is causing this crash. Is there anyway of us finding out more information as to what is going wrong? It’s only TD that crashes, all other windows apps remain running without issue. We’ve put a lot of things in place to monitor and catch hardware changes such as using the Monitors.Dat and the Video Devices.dat. We’ve essentially resulted in building a watcher app for the time being to check when TD stops running and rebooting it again into the correct.toe file. That’s not ideal though as it’s a live environment and it silently crashes at sporadic intervals - again we’ve logged and check this for patterns - there are none!
Any thoughts on this would be gratefully received.
We launched the live project on version 2023:12000 and have also since tried 2023:12120 just to be sure. It’s a tricky one as we invested a lot of time in testing off site in a one to one mock up and didn’t experience any of these issues.
Sorry you’re having these problems. A crash without a dump file often indicates it’s an external library that is forcing Touch to close, but its hard to say for sure.
It is also possible to not have a dump file if the machine is running out of resources and there is no memory available to create the dump file. So it may be worth checking the Commit size of the process in the task manager.
What type of processor are you running the machine on? We’ve had a lot of users run into hardware failures on the 13th and 14th generation Intel i9 processors that lead to random crashes. These generally create dump files, but its hard to say what effects this could cause.
Some other users have used an online utility called Prime95 as a stress tester to look for hardware faults.
You could also try setting the environment variable TOUCH_TEXT_CONSOLE = 1. This will open a text window alongside TouchDesigner and the Orbbec library may print additional error messages to that window that could be useful.
Thanks @malcolm. It’s a AMD Ryzen 9 7960X. The odd thing is we have exactly the same PC build in our studio and this one doesn’t fail. This setup has always been very reliable for us and has become a bit of a standard.
When you say “You could also try setting the environment variable TOUCH_TEXT_CONSOLE = 1.” this sounds useful. I’m not sure I’ve ever done this before, so where do we set the environment variable - is this within preferences for Touch?
Lastly your note about an external library has made me think - we did possibly swap the .dll for the Orbbec to a later version on this machine - i’m now wondering if that could be causing these silient crashes. This was done as we found early on that the Orbbec’s were lagging for no reason. This was later found to be solved by updating the onboard firmware of the Orbbecs to one before latest release. I think we swapped the .dll during that process of trying to resolve that. Since then though we have installed 2023.12120 so not sure if the .dll would have been restored to the standard one during that installation process. I can’t remember off the top of my head whether that .dll sits in the TD directory or externally to it under the Orbbec SDK…
We did try the WinDbg route early on, but as it can take a day or so for the crash to happen and it’s not in a live environment, the file size just became too huge and it affected performance, hence why we have tried to fully recreate this in the studio with the same build, hardware, etc.
What feature did you use of WinDbg? There shouldn’t be any file created until the crash occurs and you create the .dmp file. Were you using the ‘recording’ feature? I wouldn’t suggest using that since as you say, the file output from that mode gets too large.