Hello,
I am not entirely sure about this memory of mine, but I feel like there was some mention about “git compatible toe / tox files” long time ago. Therefore I just wanted to ask if this is planned for the future? Thanks 
3 Likes
Yes, this is something we’re looking at, but I don’t have a timeline or plans to share at this point. If you’ve got any thoughts or requests on this, feel free to post them here and we’ll keep them in mind as we move forward.
Great, thank you for info.
I have one little idea that I would like to mention. I was thinking it might be cool if tracked toe / tox files would show git related status information on operators and parameters. If operator or parameter was added or modified since last commit, some small indicator might be nice.
I am attaching visualization in vscode as inspiration (see yellow M and green A).
I think this would be nice addition to bold non-default parameter RFE that I tried to mention at the bottom of this post.
1 Like
Direct Git integration is on our wishlist, but might not necessarily be in an initial release of the file format change since it requires some UI work that is dependent on other projects.
1 Like
Sure, thank you very much for info 
I don’t know if it is even possible but
it is pretty neat that with files that are text based witch makes it possible to look at git comits and diffs
fx KiCad
yes, the idea is to have a text based mode to allow for merging and diffs. It’s somewhat possible now if you try the toecollapse and toeexpand utilities that are in the bin folder, but the idea is to make it a little cleaner and more seamless than those utilities do right now.
2 Likes
I have been thinking about binary data stored in toe / tox files (like for example locked TOP operator storing texture in a scene file). Please may I ask how are you planning to save such data when it comes to text based files (if there is already some plan on what to do with such data)?
(I am just curious whether they will be visible as jibberish in diffs, or is there some trick to hide them somehow?)
The common way for storing binary data in text is using Base64 encoding ( Base64 - Wikipedia ). It is essentially jibberish, but it uses a subset of characters so that it displays in text editors fairly cleanly and doesn’t interfere with formatting.
This gets used whenever data has to remain binary like if you lock TOPs, use the virtual file system, or have file protection (encryption) enabled .
1 Like
Would it not also be possible to simply save stored data in it’s own binary file? This way we could easily exchange stored and can maybe update stored data without putting dirt on the logic-part of a file.
Another question would be how granular you would do the setup? Having every node as its own file would pollute everything, but the whole project as one bix ol textfile is also not ideal. I feel that every COMP should simply have its own file and maybe subdirectories when there are nested comps.
I feel it would be good to only store non default parameter values (even though when they change later on that might get tricky).
Also, this could simply be like a word file, which in itself is simply an archive that can be unpacked with 7ZIP or similiar (which I find kinda funny tbh).
1 Like
We’re looking at multiple ways to split the files right now and likely it will be configurable to some extent. Using a single file will definitely still be an option as it is now, but to reduce merge conflicts and make reading the files a little easier there will likely be one or more modes such as file-per-node, or file-per-component.
Splitting the data portions out on their own is an interesting idea that we haven’t explored much yet beyond the existing use of external files with dats and file in nodes. Its definitely something that would be compatible with the system though, so it could be an option at some point.
Our current system only saves non-default parameter values and that will still be the way going forward. We have internal systems already in place for when we change a parameters default (it doesn’t happen often, but there have definitely been cases).
Our current toe/tox files and the binary versions going forward already do use a compression system to reduce file size. They are not directly readable by 7zip because of header data and a few other differences, but we intend to continue with tools like the existing toeexpand/collapse utilities to let them be parsed outside of touch.
3 Likes
This is super interesting, thanks for sharing all info.
I mentioned this at the last summit, Flash did the same thing at some point. They had a binary FLA old way or the newer xml based way .xfl format . Flash files had graphics and other binary assets and I think what they would do is not store them in the xml but reference them and store them in a lib folder next to the xml file.
Here is a basic Flash file with a banana bitmap asset in the library.
test.zip (269.0 KB)
2 Likes
This will be needed with larger projects or a lot of binary data embedded in the ‘code’ when working with git repositories, since providers usually put an upper bound on repository size and the recommended path to scale is via git-lfs. Please allow binary blobs to belong in a specific spot in the filesystem that can be filtered with git-lfs. As for single-file or text-only solutions, base64 is on average 33% larger for the same payload, also putting a damper when version controlling the stuff.
1 Like
I do like the idea of externalizing binary data for some cases, but out of curiosity, what binary data are you storing in your files? There are only a few things that get stored as binary blobs like locked TOPs or python storage.
Sometimes .DLLs in VFS, 3D textures (would love for Moviefilein to support 3D textures in DDS). I do consider having a lot of binary data in the network to be an anti-pattern but sometimes crunch-time collaboration / last minute stuff makes you do weird stuff.
Edit: Privacy-enabled TOXes sometimes.
What is the state with the new file-format anyway? We got a little overview during the event in Berlin, but there were some unanswered questions as the level was “We are not sure yet”.
My main question still is how granular the file will be. At least from what I gathered, it seems like to still only save a single file. Without tooling provided by derivative we are still in the same spot as before (more or less) in having to take care of externalisation ourself.
Are there plans automate the “expansion” of a project without us having to resort to manually saving the toxes. I suppose the toxes being textbased helps in git ignoring saved toxes that did not actually change anything. (COMPs being flagged dirty when externalChildCOMPS are edited makes autosaving all dirty comps pretty hard).
IMHO Idealy we would get something like one folder + one json per COMP, in the folder being all childCOMPs etc. (at least fo externalized COMPs.)
On another note, I am not a big fan of using the same file-extension for the new JSON-Based and the original (compatbilitywise). I think we should get a new fileformat (.jtox, jtoe ?) because I feel like thiss will get extremy frustrating!
It is still a work in progress. We haven’t really gotten into serious testing yet, so it’s hard to give a timeline on how much work needs to be done.
It will definitely have options to be more than just a single file. We currently have 3 modes we’re testing: single file, file-per-network, and file-per-node. The split files are nested in folders so that children of a network will be inside a folder that matches the parent name. Externalizing binary blobs may be a further option that could be compatible with any of those modes.
The split file modes can work in tandem with manually externalized toxs so that you can have a split project file with single file toxs, or vice versa.
The main difference between the two modes (split file vs external toxs), is that the split file acts identically to a single file once it is loaded into TouchDesigner. So for example, in the upcoming release, nodes in external toxs can use file paths that are relative to their external tox file location, but all file paths in nodes of a split file are still relative to the project toe file location. This is so that you can switch between single/split files at any point without it affecting the project. For example, automatic backups and crash recoveries of a split file will be saved as singe files and will still work like the original file when opened.
We’ve gone back and forth on whether to use a new file extension or not, but are currently sticking with toe/tox. Is your primary concern accidentally loading new files into older versions of touch? We don’t technically support doing this now, but we realize it still happens because it frequently appears to work (although often the problems are just hidden). The new files will obviously not be compatible at all with older versions of touch.
It gets a little complicated because the new files actually have multiple subformats including both text and binary versions as well as a few variations of encryption/password protection. We’ve also found that, whether we like it or not, tox has become synonymous with an external component so changing that could lead to confusion. But that said, Microsoft made a similar transition from .doc to .docx
10 Likes
@robmc is there a timeline available at this point? It would be great to have a better workflow with Git instead of just committing a binary file with ‘multiple changes’. Thanks!
While we wait you may want to look into this: