Type hints / type checking in TD

I’ve been using pydantic for type hints recently and just the warnings from the linter alone have been super helpful, not to mention the runtime errors.

I’ve been on the type-wagon for a while now and recently completed some very large TD projects that would have benefited enormously from more complete type checking integration in the TD Python world.

Is anything like this on the roadmap? I can only imagine there’d be similar benefit for the Derivative as well as to the users for the undertaking.

There are two ways at the moment:

  1. TD-Completes me: GitHub - picturesbyrobots/td-completes-me: A Simple Auto Completion Engine for Touch Designer
    In general this is amost the best solution as this does typeinferince based on the actual project and also supports extensions. I just have issues getting it to work reliable.
  2. TD Stubs by @tekt https://github.com/optexture/td-components/tree/master/lib/_stubs
    Download them and place them in a folder called typings. You need to open the project via vsCode from the projectRoot, but then they ae a delight. You can see this setup working in action here:
    GitHub - AlphaMoonbaseBerlin/TD_RenderInteractionFramework
  3. I also work on some stubs by parsing the Wiki, but they still have quite some work, so nothing final (but looking for contributors :wink: )GitHub - AlphaMoonbaseBerlin/Python_TouchdesignerStubsFromWiki
1 Like

Like @alphamoonbase pointed out there are some great places to get started on the TD side of the world. These days we’ve been pushing hard on how we organize our own python and extensions to allow for autocomplete for any of our studio authored code base. That’s a little easier to wrangle. Happy to share some ideas if that’s something that’s interesting to you.

1 Like

@raganmd I would definitely be interested in hearing how you organize your files :slight_smile:

1 Like

Sweet thanks for the leads everybody.
@raganmd would love to hear how you’re approaching this.
@alphamoonbase those @tekt stubs were definitely helpful for quieting down the warnings

Still wondering if Deriv has plans to take this into the product, though… doesn’t this feel like it’d be a win-win for internal and user development?

1 Like

@hardworkparty and @monty_python - sorry for the delay, we’ve been doing a little internal re-org for our practice and I wanted to share something that was closer to a better reflection of our current practice.

Disclaimers

  • This doesn’t address stubs from TD, and instead is a focus on how to better auto complete the code you author as extensions.
  • This doesn’t work for all projects, and doesn’t work for all developers.
  • This does require a little bit of Python and TD wrangling to get this all behaving, but has been worth it to us.
  • This isn’t as portable as would be ideal… in a perfect world it would be easier to pluck one of these subcomponents up and drop them into another project. In our case after a lot of internal discussion we settled on the reality of working with a base project structure that needed better auto-complete support as our primary focus.

A sample project file can be found here:

Extension Auto-complete

Detailed write-up

Coming soon

Overview

Our primary objective here was to help solve our big challenge of Python interfaces authored by two different devs that needed better auto-complete support. This is pretty straightforward in a pure python context but is more difficult a Python class belongs to a TouchDesigner op.

Our biggest changeover was to find a happy solution between the behavior of local/modules and how an editor like vsCode understands Python libraries. Our team largely uses vsCode but in theory this same idea should work in an editor that has similar mechanics.

Python Type Hinting

The real magic for us really came from Python’s type hinting. If you’re unfamiliar, the structure of this looks something like:

def foo(val1: int, val2: int) -> int:
    return val1 + val2

A type follows a colon after the argument. This works for any default type, but is also valid for any of your own objects. For example:

class TDFoo:
    def __init__(self):
        self.name = "Derivative"
        pass

def TDTest(my_object:TDFoo) -> str:
    return my_object.name

Here we can create a custom class, and use that class as a type hint for a function. That’s all well and good, but why that’s really helpful is because your code editor can use that hint to autocomplete for you:

Gotchas

The tricky part of this is that both your code editor and TouchDesigner need a way to resolve your Python use. In vsCode that’s not too wild, but in TouchDesigner we need to be a little more creative. For the sake of simplicity, we’ve moved all of our class objects to local/modules we’ve done this in part since it’s easy to access Python modules in TouchDesigner if they live in this path with the pattern mod.YourDATName.YourFuncOrClass. This separates your extensions from the Ops that use them… which is not ideal, but for a sufficiently complex Python implementation it’s been worth it.

vsCode Set-up

In a code editor this doesn’t look too unfamiliar. We tend to treat operators that do big work in our projects as singletons - we might have an op called DATA that is the Data store for our project, and we only have one of them. In these cases we tend to treat the Python for that op as it’s own library. In that case we create a directory for that library and include an __init__.py file to create a single instance of the class we’ll be using as a singleton:

Where the magic happens is in the use of another Python library we typically call lookup:

lookup allows us to create a variable that points to an op with a global op shortcut, but is hinted as the class object used for its extension.

Rather than using an op.someGlobalOpName pattern to access these ops we instead locate that operator through our lookup module. Which means we now get auto-complete for any of the methods that belong to that object:

TouchDesigner Implementation

To make this magic work in TouchDesigner, we have to do a little op organization in local/modules. Here we can add each of the python files that contain our class objects (skipping the __init__.py file which is really our helper for our code editor). The only exception is that we need to pull in the __init__.py file for lookup, but put it in a DAT we name lookup. TouchDesigner will allow us to import a DAT by name - so while our code editor will resolve the directory called lookuplookup/__init__.py, we have to help TouchDesigner complete that same idea.

If we look at these side-by-side you can see how this might look in vsCode and in TouchDesigner:

Takeaways

We tried several different permutations of this before landed on this particular implementation… and I can’t say that we wont change in the next year if we find something easier. That said, it’s been a huge benefit to have code completion between extensions for us. So far, despite being a little cumbersome to set-up, it’s been worth a little bit of juggling.

Also a huge shout out to @ishelanskey as he’s driven a lot of our exploration and kept the fires burning for better code completion in our projects.

@Ivan - I’m also going to tag you in here. I know you’re always interested in how the community is making python work in TouchDesigner.

3 Likes

@raganmd thank you very much for detailed write-up. I think I get the idea and it is an interesting approach to this problem. Up until now I haven’t seen this kind of structure in TD, but it makes sense in a situation where autocomplete is necessity.

Thanks as always for sharing so thoroughly and documenting so clearly on this. This is clever and solves a problem I didn’t realize I hadn’t contemplated, which is type hinting (and the completions that go with it) for extension classes at the top level.

I agree that it’s a little clunky to collect the extension DATs into local/modules if you intend to move extensions into other projects in the future, but outside of that, if you’re using VSCode to edit, the location of the DATs is irrelevant. Like you, I generally use extensions as singletons so this pattern would work well for larger projects, especially with multiple developers mostly keeping to their own domains.

Thanks again for this.