We have a project running on a remote site where TD silently crashes with no logs or any trace as to why. We’ve tested the same project here in the studio with the identical setup and the same hardware connected, and we don’t get this result, so it seems to be something local to the environment. We have 3 Orbbecs Femto Bolts running and 2 x 1200 x 1080 projectors for the video output connected, using a RTX 4090 card.
My question is really more general in looking for some guidance as to how we can trace and understand what is causing this crash. Is there anyway of us finding out more information as to what is going wrong? It’s only TD that crashes, all other windows apps remain running without issue. We’ve put a lot of things in place to monitor and catch hardware changes such as using the Monitors.Dat and the Video Devices.dat. We’ve essentially resulted in building a watcher app for the time being to check when TD stops running and rebooting it again into the correct.toe file. That’s not ideal though as it’s a live environment and it silently crashes at sporadic intervals - again we’ve logged and check this for patterns - there are none!
Any thoughts on this would be gratefully received.
We launched the live project on version 2023:12000 and have also since tried 2023:12120 just to be sure. It’s a tricky one as we invested a lot of time in testing off site in a one to one mock up and didn’t experience any of these issues.
Sorry you’re having these problems. A crash without a dump file often indicates it’s an external library that is forcing Touch to close, but its hard to say for sure.
It is also possible to not have a dump file if the machine is running out of resources and there is no memory available to create the dump file. So it may be worth checking the Commit size of the process in the task manager.
What type of processor are you running the machine on? We’ve had a lot of users run into hardware failures on the 13th and 14th generation Intel i9 processors that lead to random crashes. These generally create dump files, but its hard to say what effects this could cause.
Some other users have used an online utility called Prime95 as a stress tester to look for hardware faults.
You could also try setting the environment variable TOUCH_TEXT_CONSOLE = 1. This will open a text window alongside TouchDesigner and the Orbbec library may print additional error messages to that window that could be useful.
Thanks @malcolm. It’s a AMD Ryzen 9 7960X. The odd thing is we have exactly the same PC build in our studio and this one doesn’t fail. This setup has always been very reliable for us and has become a bit of a standard.
When you say “You could also try setting the environment variable TOUCH_TEXT_CONSOLE = 1.” this sounds useful. I’m not sure I’ve ever done this before, so where do we set the environment variable - is this within preferences for Touch?
Lastly your note about an external library has made me think - we did possibly swap the .dll for the Orbbec to a later version on this machine - i’m now wondering if that could be causing these silient crashes. This was done as we found early on that the Orbbec’s were lagging for no reason. This was later found to be solved by updating the onboard firmware of the Orbbecs to one before latest release. I think we swapped the .dll during that process of trying to resolve that. Since then though we have installed 2023.12120 so not sure if the .dll would have been restored to the standard one during that installation process. I can’t remember off the top of my head whether that .dll sits in the TD directory or externally to it under the Orbbec SDK…
We did try the WinDbg route early on, but as it can take a day or so for the crash to happen and it’s not in a live environment, the file size just became too huge and it affected performance, hence why we have tried to fully recreate this in the studio with the same build, hardware, etc.
What feature did you use of WinDbg? There shouldn’t be any file created until the crash occurs and you create the .dmp file. Were you using the ‘recording’ feature? I wouldn’t suggest using that since as you say, the file output from that mode gets too large.
I was wondering if your issues has been resolved yet?
My installation seems to be having similar problems, closing without warning or .dmp file.
I am also using a Femto Bolt, which we updated while prototyping/installing and I am wondering if this could be the case/solution?
I hope you can help me out, I am on my way to the museum now trying to pin point the culprits.
Will send some more details later, just in case, you all could help me out.
We’re still looking into your issues, but so far none of the dump files you’ve sent point specifically towards the Femto Bolt. The dump files actually show a variety of places which usually indicates some sort of memory corruption is happening either due to a software bug or a hardware fault.
Thanks for ruling this out, and the effort overall.
I haven’t been able to reprocess the silent crashing yesterday on site, I’ve relayed the technical issues to the client. They have someone an IT specialist in house, that can look into the hardware.
Ok, I’m experiencing very similar behaviour in 2023.12230. Sudden silent crash while edditing network, no logs or error message. It is very frequent and I am beginning to worry about current stability of Touch with kinect/orbbecs.
Crashes are occuring when working with particular part of the project - in current player facetracking using kinectazureCHOP and facetrackCHOP from kinect rgb camera (kinectazureselectTOP). Wanted to test both systems, but a lot of silent crashes (5 in 30 minutes) occured, while working on facetrack component.
Please advise how can I properly report this bug. will project file help, or should I try to use the winDbg while reporting?
Touchdesigner 2023.12230, RTX 3090, Ryzen 5900X, 32 gigs of ram, windows 11, kinect azure
Sorry you’re having problems. You can attach the WinDbg tool to a running instance of TouchDesigner to capture crashes that touch doesn’t get itself. We have some information on that here: Using WinDbg to Debug Crashes - Derivative
It does feel like we are seeing more stability issues with the Nvidia FaceTrack component recently that are happening inside the nvidia library. Unfortunately we can’t tell why these issues are happening, but Nvidia hasn’t updated that package in 2 years now so its possible there are compatibility issues with recent drivers.
If you can send us any dumps from WinDbg or if we can recreate it from your project files we might be able to confirm at least whether it is an orbbec, face track or some other issue.
Unfortunately, there is no indication the crashes you are getting are related to the Nvidia drivers. So the issue you are having is occurring elsewhere.
Yeah, the crashes still occur, will run WinDbg
[some time later]
I’ve managed to save the .dmp with a WinDbg command: .dump /ma C:\example\mydump.dmp and the file is 8 gigs. Here’s the link: