Fatal (process suspended in crash handler) after an unhandled C++ STL exception inside a cook on the main thread when a CPlusPlus CHOP's output is rebuilt (getChannelName/setString) one cook after its first connect — cook-order dependent

Bugs
, ,
1

Build: TouchDesigner 2025.33070 (64-bit, Commercial), Windows 11 Pro, Python 3.11 in-process, NVIDIA RTX 4070 Ti (driver 560.94). Also seen on a second machine? No — only on the machine with the lidar hardware attached (see Isolation).

Summary

TouchDesigner stops cooking permanently (main thread parked in TD’s fatal-error handler, no dialog visible in perform mode, no Windows Application-log event, process still alive so a watchdog must kill it). Minidumps taken while parked show, on the main thread, an unhandled C++ exception thrown from the MSVC STL (msvcp140!_X*_error → VCRUNTIME140!_CxxThrowException → terminate/abort → libTD handler → wait) with libTD cook frames beneath it, and beneath those a normal Python callback (PyObject_GetAttr / _PyEval_EvalFrameDefault / _PyFunction_Vectorcall, in one case with numpy _multiarray_umath frames) — i.e. a Python callback touched a TD object, that pulled a cook, and something inside that cook threw a C++ exception that nothing caught.

The throw site is the same in all three dumps (return addresses, libTD.dll of build 2025.33070):

msvcp140+0x94020 / +0xb656 / +0x94400 / +0x943b0 / +0x94090 / +0xc682   (STL throw helpers)
VCRUNTIME140+0x5acd                                                       (_CxxThrowException)
libTD+0xd9615e  ← libTD+0x120c701  ← libTD+0x1ddfd07  ← libTD+0x1de01f0  ← libTD+0x1239301
libTD+0x1e88dc6 ← libTD+0x1e85720  ← libTD+0x520e3d8                      (cook chain)
handler: libTD+0x47de290 / +0x47de2aa, then KERNELBASE wait

No frame of our C++ plugin is on the main thread in any dump; the plugin’s own threads are in ordinary waits (SerialPortChannel::open → CreateFile, and a named-mutex wait).

Setup (what runs in the process)

  • A CPlusPlus CHOP (Slamtec RPLidar S2 driver, open source fork, SDK 2.x) that connects to two USB-serial lidars from a detached std::thread started from execute(); the cook thread never blocks on it. Precision 8 → 2880 samples × 4 channels, cooking every frame. Downstream: Script CHOPs with numpy callbacks, an Execute DAT reading numpyArray() every frame.
  • An embedded MCP/HTTP server (FastMCP + uvicorn) started from a Python extension in a threading.Thread; a parameter change on that COMP stops and restarts it (Stop → close listening sockets → Start() again ~30 frames later via run()).
  • At boot our config extension (Python) disables and re-enables that server about 4–5 s after project load; the lidar CHOPs get Active=1 in the same second, so the first device connect and the server restart overlap.

Reproduction (4/4 on the production machine)

  1. Load the project (the lidar hardware attached, ports COM3/COM4).
  2. Frame ~14: Embody starts the HTTP server thread on port 9870. Frame ~26: our extension turns the server off and on → the extension’s new restart path closes the old sockets and schedules Start() 30 frames later.
  3. Frame ~20: Active=1 on both lidar CHOPs → their execute() starts the connect threads (CreateFile on COM4).
  4. Frame ~56–61: the delayed Start() runs (new uvicorn thread, registry file write) — the main thread never returns: last log line at frame 61, heartbeat file stops, the connect log stops after CONNECT COM4 (the connect thread is parked in CreateFile, which is consistent with the whole process being suspended by the handler).
  5. Dump taken 5 min later shows the stack above.

Isolation

  • Same .toe, same DLL, same Python, but the HTTP-server extension WITHOUT the new restart path (older revision of that extension’s DAT): boots clean, 2/2 today, dozens of times over the past weeks.

  • Same new extension, no lidar hardware attached (serial port absent or a port with no device so the connect fails in getDeviceInfo): boots clean (developer machine, 2/2).

  • Same new extension on two other projects in the same process family (no CPlusPlus CHOP): boots clean.

  • The identical exception signature happened once more on a 12.7-hour-old process at the moment a Python extension was reloaded from disk and restarted the same server while the lidars were streaming (numpy frames under the cook chain in that dump).

  • Wall test T1 (18:23): same new extension, but the HTTP server is disabled by our config 3 s after boot (so the sequence is: server started at frame 13 by the project’s own setting, stopped at frame 24, lidar CHOPs enabled at frame 24, first CONNECT COM4 ~1 s later) → freezes identically (dump …182836). So a stop/restart of the server is not required; a start followed by a stop shortly before the first device connect is enough. No MCP/HTTP client was connected during any of the boots.

  • Bisect on the .toe itself (evening of 12-09, 12 boots on the production machine): the good project file (saved 09-09) re-saved on the dev machine without any change freezes (T1f). Expanding it with toeexpand, collapsing it back unchanged with toecollapse and booting is clean (H0), so we can transplant single file groups from a freezing save into the good one: time-slice state of 13 CHOPs → clean (H1); Script CHOP / Timer / table state → clean (H2); the .n node files of /project1 (only its pickled storage dict differs: two integer generation counters and one float key removed) plus five DAT/SOP viewer positions → freezes (H3). None of those values has any code path to the CPlusPlus CHOP; the failure is therefore sensitive to load-time layout/ordering, not to the meaning of the data. Further halving (storage-only vs viewer-only vs single keys) is in progress.

  • With the plugin’s map file, the frame SlamtecCHOP_V3.dll+0x2108d present on the main thread in 6 of 7 dumps is SlamtecCHOP::getChannelName()+0x12d, i.e. the return from OP_String::setString(). The throw happens inside libTD seven frames below that call. Our getChannelName only reads a member and calls setString. It is invoked by TD when the CHOP’s output is (re)built — which in our plugin happens one cook after a 60-cook startup delay, because only then it reads its Precision parameter (720 → 2880 samples). That cook coincides with the first device connect. Two negative controls on the dev machine (no lidar): forcing an out-of-range coordinate enum (empty channel names at the first build) and forcing the other enum value (channels renamed angle/distance → x/y at the rebuild) both boot clean, so the plugin-side content of the names is not sufficient to trigger it.

  • Verified: TD reloads a DAT with syncfile from disk at project load even when the file is older than the .toe, so baked DAT text differences are ruled out.

  • Final halving (21:10): the single difference between the good and a freezing project file is one stale key in the storage dict of /project1 (beet_tap_until, a float timestamp). When present, our Python onFrameStart on frame 1 logs a warning and calls unstore() on it (which dirties the dependents of that storage) — and the project boots. Remove only that key: freeze. Put it back: boots. So the crash depends on cook ordering around the moment the CPlusPlus CHOP’s output is rebuilt, not on any data.

  • Positive confirmation (22:04): take the freezing project (no stale key, original plugin build that froze it four times) and add nothing but op('/project1').store('x', 1); op('/project1').unstore('x') in onFrameStart for frames 0–2. It boots cleanly, both lidars connected, held for 180 s. So the minimal toggle between freeze and boot is one store()+unstore() on the root component during the first frames — no data, no parameter, no plugin change.

  • Plugin-side avoidance confirmed on the wall: a build that reads its Precision/coordinate parameters from the very first cook (so the output is never rebuilt after the startup delay) boots the same freezing project cleanly with both lidars connected. The rebuild after the delay was the trigger point. The underlying exception in libTD remains; we simply no longer hit that path at that moment.

Attachments

Note: the raw .dmp MiniDump binaries were not retained on our local disk past the day they were captured; only the text reports we generated from them the same evening survive. Those are attached in full below — the main-thread stack (the part relevant to this report) is unaffected by working from the text form rather than the binary.

  • Freeze at 17:13:10 — lidarman_20260912_171310.txt (annotated summary), lidarman_20260912_171310_alle_threads.txt (full thread dump), lidarman_20260912_171310_hoofd250.txt (main/crash-handler thread, 250 frames, symbolized).
  • Freeze at 17:18:49 — lidarman_20260912_171849.txt, lidarman_20260912_171849_alle_threads.txt, lidarman_20260912_171849_hoofd250.txt (same three-file set).
  • Freeze at 17:52:37 — lidarman_20260912_175237_hoofd250.txt (only the main-thread excerpt was kept for this one).
  • Freeze at 18:28:36 (fourth occurrence, T1, see Isolation) — lidarman_20260912_182836_hoofd250.txt (identical main-thread chain to the three above) and lidarman_20260912_182836_modules_libTD.txt (the module-table export the numbers below are drawn from).
  • Module table: libTD.dll size 150,974,464 bytes, base 0x7ffb98430000, PDB TD.pdb/4BF1537E3A8245AD9E3B3EE5D5CBC40F1 (build 2025.33070); python311.dll 5,750,552 bytes (TD’s bundled 3.11); msvcp140.dll/VCRUNTIME140.dll from the TD install.

Questions

  1. What does libTD+0xd9615e (called from libTD+0x120c701) throw, and under which condition? We suspect either a std::out_of_range/length_error in operator or channel bookkeeping while a CPlusPlus CHOP’s output is changing on its first successful connect, or a std::system_error from a threading primitive being re-entered.
  2. Is it expected that a C++ exception raised inside a cook that was pulled from Python (attribute access / numpy buffer) escapes to std::terminate instead of becoming an operator error? A Python-level error or an op error would be recoverable for us; the silent suspend is not (the process stays alive, so no crash dialog, no WER event, and a watchdog is the only way out).
  3. Is there a known interaction between a Python socket/asyncio server being closed and reopened from the main thread and CPlusPlus operators cooking in the same frames?

Happy to run an instrumented build or provide a reduced project; the plugin source is public.

freeze_lidarman_20260912_171310.zip (40.4 KB)

2

Hey,

Sorry that you’re having this issue. Could you try upgrading to 2025.33230 and test there? Several fixes were made in that build.

Additionally, could you create a spindump via the task manager and send those over, since we can read them and understand where internally it maybe failing. The dmp files may be large, but they compress well, and can be stored and sent over compressed.

Could you also point to the plugin source.

Varad