Bug Report: Inconsistent Extension Re-initialization in TouchDesigner with Temporary Files and Indirect DAT Editing
Summary:
Extensions in TouchDesigner do not consistently reinitialize when edited through various indirect methods such as external editors using temporary files, or from editing one DAT’s contents via another. The changes to extension code do not properly take effect until the project is saved and reloaded, which creates significant challenges during development.
Steps to Reproduce:
- Create a new project in TouchDesigner (TD versions tested: 11880 and 11600, but seen as far back as 2022).
- Add a custom component with a Python extension.
- Edit the extension using any indirect method, such as:
- Temporary files opened in an external editor (i.e., not synced files).
- Modifying the content of one DAT from another DAT (via Python code or DAT references).
- A custom component or tool that edits the content of a DAT indirectly (such as using websockets to update a DAT’s contents).
- When using an external editor, save the changes to the extension.
- Reinitialize the extension using the “Re-init” button on the component.
- Observe the behavior of the extension, particularly whether new functions or updates are applied.
Expected Behavior:
The extension should reinitialize fully after any modification, and all changes (including new or altered functions) should be reflected without the need for a project reload. Pressing “Re-init” should force the extension to fully refresh.
Observed Behavior:
- The extension does not always reflect updates, even though the debug output indicates it has reinitialized.
- Some old references to functions persist in memory, despite deleting or altering the extension code.
- Even after deleting the code in the extension, some functions continue to work, while others do not update correctly.
- The issue persists across different machines (Windows and macOS) and different projects.
- This behaviour is inconsistent, but will occur in the outlined situations, even if not immediately
Potential Cause/Insights:
- There seems to be an issue with caching and propagation, where older references to the extension’s functions or code are retained in memory.
- This issue appears specifically when indirect editing methods are used, such as:
- External editors using temporary files (not synced files).
- Modifying the contents of a DAT indirectly from another DAT.
- Custom components or external tools that update a DAT via methods such as websockets (in my case, my custom TDCode component).
Key Insights from the TouchDesigner Discord Discussion:
- @alphamoonbase.berlin:
2.“I sometimes do have an issue where I would need to cut and paste a textDAT because it somehow was holding an older reference to the extension.”*
3.“Mayve it is some moduleCaching and cooking propagation that is creating an issue.”* - @noah | hard work party:
5.“Are you editing the extension files externally? If so, are your extensions saved as external text files or are you using the temporary buffer that TouchDesigner creates?”* - @shamanic (me):
7.“I 100% have seen this issue before & not with my tool. I think the issue is deeper than just my tool’s indirect editing, but the fact that you seem to sync files over editor & never have issues vs I have always edited with temp files feels meaningful.”*
8.“It may just be that devs that work with extensions and sync to an external editor are less likely to encounter these issues.”* - @noah | hard work party:
10.“I completely do not trust that buffer thing. Way too many failure modes. Never use it for anything more than a quick par exec or some kind of short callback thing.”*
Workarounds:
- Synced files: Using synced external files avoids this issue and is the preferred method for external editing.
- Direct DAT editing: Editing directly in the TouchDesigner DAT editor also avoids the issue.
- Save and reload: Manually saving and reloading the project is a temporary workaround, but it disrupts workflow and slows development.
Additional Context:
This issue has been observed across different versions of TouchDesigner, multiple operating systems (Windows and macOS), and on multiple machines. It affects indirect editing workflows, especially those involving temporary files or external tools that indirectly modify DAT contents (such as via websockets or other non-DAT editor approaches).
It is particularly relevant for my custom tool, TDCode (not yet publicly released), which relies on websockets to update DAT contents. While I have no control over this bug, it’s critical that this functionality works reliably for my tool to be effective.
Request:
I recommend a deeper investigation into how TouchDesigner handles edits made through indirect methods (temporary files, inter-DAT updates, websockets, etc.). It would be helpful to explore how module caching and cook propagation affect the re-initialization of extensions.