Tox workflow inspired by Houdini

Hello,
I think it might be really cool to consider following tox workflow for some future improvements.
I like the way Houdini is managing its hda’s and I thought it might also be a way to go for Touchdesigner.

In Touchdesigner are toxes referenced by their absolute or relative (relative to project file) paths. This causes some problems in more complex situations (moving project (or toxes) to another directory breaks relative paths to toxes - requiring re-save with correct path in external tox parameter, etc.)

On the other hand, Houdini builds Asset Library by scanning all paths provided in HOUDINI_PATH environment variable - making it much more flexible setup. One could easily setup and maintain per-project / per-version / per-user environments (just by using .bat launchers with specified paths to folders that should be scanned for hda’s). This makes it super easy to change paths to hda’s - just by editing the environment variable. Once scanned, all hda’s are available among nodes - you could just hit tab, write name of your tool and add it to project (which is quite nice in my opinion). I feel like this system would be great also for Touchdesigner.

With this setup also tox versioning sounds much simpler as could have your tox releases versioned in separate folders and reference specific version through this environment variable. :slight_smile:

Hello,
please may I ask if there are some bigger tox workflow improvements planned for 2023 experimental branch?

Up until now I have been using custom “asset manager”, which was my take on a super simplified asset management (inspired by Houdini’s asset management). I am now thinking about backporting it to some older projects to make the maintenance easier. Therefore I thought I might first ask if there are some plans to upgrade default tox workflow before I dive into this :slight_smile: Thanks.

Some upcoming small additions to make it easier to work with external files will be in next experimental.

Not related to those changes, are you familiar with project.paths? Project Class - Derivative
It needs a UI and workflows built around it but was meant to help manage project paths of all sorts.

@ben thanks for info. Unfortunately project.paths doesn’t help much in this area… I mean it looks like a nice concept, but I am primarily looking for a way to automatically configure component paths from environment variables. Configuring them manually with such prefix (and then providing a json for every single file) is an option to a certain extent, but it doesn’t allow for the level of flexibility TD needs in larger pipelines.

So far I have been trying to get this kind of workflow in TD:

Lets say I am developing various td modules (lets call them packages) in their respective git repositories. These are then being deployed (either using CI or manually) to one network location available for all users / servers / render nodes. Its structure could be as simple as this (names are made up just to illustrate this workflow):

/packages
├── td_utils
    ├── 1.4.0
    ├── 1.4.1
├── td_rocks
    ├── 1.0.0
    ├── 1.1.0
├── td_trees
    ├── 3.0.2
    ├── 3.1.0
├── td_forest
    ├── 2.1.0 (depends on td_rocks-1.0.0 & td_trees-3.0.2)
    ├── 2.2.0 (depends on td_rocks-1.1.0 & td_trees-3.1.0)

Each version contains stuff necessary for given package to work - such as tox files, scrips, shaders, assets, unit tests, etc. One could then handle whole pipeline just by using environment variable TD_PATH, that would contain paths meant to be searched for toxes. This setup makes it easy for a project to define package dependencies (with specific versions being targeted).

For example project foo would define its dependencies:

td_utils-1.4+
td_forest-2.1.0

To start this project, only TD_PATH environment variable has to be configured. Depending on complexity, this could be either handled manually in some bat script, or with something like Rez in more complex environments.

TD_PATH=/packages/td_utils/1.4.1;/packages/td_forest/2.1.0;/packages/td_rocks/1.0.0;/packages/td_trees/3.0.2

Once TD starts, it looks for toxes within these paths in TD_PATH, and if it finds some, it updates it local library of toxes. Its library could then look like this:

/packages/td_utils/1.4.1/bottom_bar.tox
/packages/td_utils/1.4.1/buttons.tox
/packages/td_utils/1.4.1/ui_panels.tox
/packages/td_forest/2.1.0/forest.tox
/packages/td_rocks/1.0.0/rock.tox
/packages/td_trees/3.0.2/tree.tox

Then it goes through all the components in the scene and loads components with matching toxes (available within this library) from disk. This way correct versions of utils and forest tox is loaded (also correct versions of rock and tree toxes are loaded {recursively} within the forest component).

In case we decide to change version of some package, it is as easy as changing dependency specification. TD_PATH environment variable is generated automatically using Rez - thanks to its dependency resolution. It is worth mentioning that Rez can also handle python package dependencies. For example if td_utils would require tensorflow-2.10.0, Rez would take care of getting it from PyPI and configuring proper environment for TD.
For simpler use-cases (where Rez isn’t necessary) we can just manually change bat script used for setting up our environment.

Long story short - as long as we have TD_PATH configured, we can start any scene or any tox without having to worry about toxes not being correctly loaded.

I guess there are many ways to approach actual implementation on TD side, but I feel like there is a need to support this kind of workflow as it allows to integrate TD in larger pipelines. Right now I am doing this with some custom component that takes care of building the library of toxes based on TD_PATH and then it goes through components in the scene, loading correct versions of toxes. However many times I stumble upon some issues that are quite hard to overcome. It would be great if this sort of workflow would be enabled by default in TD. It could also later serve in many other ways - for example such library could make it possible to extend OP create dialog by custom components. That way could user add some path containing custom components to TD_PATH and they will be right away available in OP create dialog. :slight_smile:

Please do you think some Pro support hours could help in this case? I would be glad to use mine for it, but I guess it might take much more time that I have :slight_smile: Anyway thanks for reading through this.

1 Like