Python debugger/profiler

As our projects get bigger and start to rely on extensions and python more and more it really becomes impossible to debug our code by injecting print/debug statements all over the place.

Another thing is if your python code is slow, it’s really hard to tell what exactly is the bottleneck.

These would be a very nice additions if possible! Thanks

9 Likes

This would easily cut the dev time for larger systems manyfold, lost count of the times I wished for a stack inspector and stepping through code.

2 Likes

I have used debugpy (GitHub - microsoft/debugpy: An implementation of the Debug Adapter Protocol for Python) to remotely debug a python project that it’s running inside a docker container. I don’t know if it would be possible to do the same with the python running in Touchdesigner. Maybe this would be a good starting point for Derivative to add the feature.

You can already attach a debugger to the running python process and pause, step through code and catch exceptions… however, there is no mapping between the code you can edit and the running modules, and therein lies the value, being able to set breakpoints and inspect the stack, watch and step through the code in the editor would save system builders a fair amount of time…

1 Like

can you please elaborate? How can you attach a debugger currently?

Actually, I take that back, I can’t seem to attach to the running process anymore with TD v2022.33910. I used to do it with vscode with the ‘Attach to process id’ debug script that is provided by default. When attaching I just searched for TouchDesigner in the list and the code injection gave me control of the interpreter with no source code mappings.

1 Like

I made a proof of concept that works but is not that straightforward to setup. Any help is appreciated:

Tested in Touchdesigner v2022.31030

2 Likes

OMG! That’s super exciting, I’ll give it a try in the coming days

Hey, I spent some time today looking at your project. Great job! The example works, however I wasn’t able to make my project work unfortunately. It would be great if someone from @derivative could look at it and maybe provide some insights what we’re missing here.

Maybe @Ivan has something to say? Because it feels very close to being usable, and that would be a huge win for the community!

A first-party debugging package (even if limited to vscode) would be a huge win for systems builders, agreed.

1 Like

I’m very interested in this but also couldn’t get it to work.

When I ran:

python -m pip install debugpy

It couldn’t find pip (because it’s running TD python.exe). Installed debugpy another way, which seemed to work, but then vscode didn’t have the TouchDesigner process as an option.

Going to keep trying but if you have any clarifications, that might help!

Here is a video tutorial on how to use the repo.

@Ivan did you have problems using it on your project or in my example?
Now that you mention the error with pip missing I remember that the first time I tried to do this (2 months ago) I also had that problem and solved it using a file called get-pip.py which was suggested in stackoverflow. I updated the repo’s README with the solution.

I just realised that there is “ensurepip”, which basicly will install PIP in to the python-libs of the current python env (in this case the TD-Python Install).
You just need to make sure that you set sys.executeable to the pythonExecutable when running this. Otherwise it should work fine.

I also updated TD_PIP to use the shipped whl as source for PIP, instead of downloading it via the PIP-API (and then forcing an upgrade…)

I did a very silly thing that seemed to have worked. I installed debugpy globally and then just copy pasted two folders that it created in my Python folder to the TD one. I’m not sure if I missed something by doing that, but I managed to get the example working, so I assume it worked

:frowning: k. I dug a little bit in to all of that.
Good news: No Need for debugpy ! vsCode takes care of that itself.

I looked a little bit in to your repository and was pretty confused about some stuff:

  1. Why did you comment out anything reffering Ports and Network when Debugpy is running in a network?
  2. Whats with the error-messages?
  3. Debugpy probably being async, it should not work nonblocking anyway.

In fact, the most important part is your .vscode.
in your projectFolder, create a folder named .vscode and create a file called launch.json

{
    "configurations": [
        {
            "name": "Python: Attach using Process Id",
            "type": "python",
            "request": "attach",
            "processId": "${command:pickProcess}",
            "justMyCode": true,
            "pathMappings": [{
                "localRoot": "${workspaceFolder}/modules/suspects",
                "remoteRoot": "c:"
            }]
        }
    ]
}

Rightclick in your projectFolder to open vsCode, go to “Run and Debugging”/Ctrl+Shift+D and press the play button and select TouchDesigner as the process.

TIMEOUT ERROR
Often it will need some time to ramp up. The default is 15 seconds, which might be not enough.
To give it more time, open commandline and run

setx DEBUGPY_PROCESS_SPAWN_TIMEOUT 90

Restart vsCode for it to fetch the envVariable and of you go.

pathMapping
Wwe need pathMapping to enable vscode to understand the paths of the modules td gives.
For some reason, /anythin will be understood as $DRIVE_LETTER/anything
This means, when we are on C:, /anything will be interpreted as C:/anything
For that we need to tell vscode to map $DRIVE_LETTER to the folder where our files live.
For me, making use of my PrivateInvestigator it is pretty simple. This is was the esample above is.
Only downside, we have to adjust the driveLetter by hand.
I suppose we could write a startupscript in touch that is setting this by itself via python.

Caviat
At the moment it seems the dbugger does not map from module/path to file.
This means to get real debugging, we need to ommit the .py on the end of our files. Not ideal as we are lossing intellisense and similiar. So have to see if there is a better way.

Breakpoints
Setting breakpoints in VSCODE does nothing as it seems. But you can still install debugby via TD_PIP and use the breakpoint function by hand. Not ideal, but at least something.


Happy debuggin!

@robmc Soooo. About that new fileformat? Having all dats being auto 1 to 1 mapped files in the project would make this like SOOOOOO good.

2 Likes

hey, there is no need to modify the ending of the file. It seems to be working as long as you provide an explicit mapping between file on disc and python extension object path. See my screenshot. On the other hand I got it to work properly only if first I set an explicit breakpoint as you can see on line 20 of my extension code in the screenshot.

The server for me also times out eventually, but somehow it doesn’t matter. Maybe the important part of this process is the dll injection which happens when you start listening on the python side? I don’t know.

I didn’t quite get the part about setting the timeout longer, do you suppose to start debugging in the vscode while debugpy on the td side waits for clients? Anyway, as I mentioned, everything seems to be working on my end, at least in a very rough test. I also need to try it in my real project

This will not work! We have much more then a single file. We could try to create a mapping for every file but this is def not ideal.
The approach I posted does already a 1 to 1 mapping based on paths in TD.

We do not need the debugpy server. It does nothing!

As said above. We do not need the debugpy server in TD.

The Debugging in VS Code can attach directly to TD without issues, it just takes a while to get up and running. Longer then 15 seconds. The default timeoutTime for the vsCode debugger to attach to a process is 15 seconds. We need to increase this time so vsCode does not abort the attachment process.

You have made quite a discovery here in this being possible, but your “Lets ignore this error” and “I have no idea how this works” and the all in all convoluted setupprocess prompted me to look deeper in to it.

The only thing really needed is the launch.json
Nothing else!

Your solution is to remove .py portion of extensions to get it to work, right? In that case it wouldn’t work universally I believe, because most of my extensions are in local/modules and as long as I don’t have same folder structure on disc simple removal of .py wouldn’t work. On the other hand I think that can be relatively straightforward to create a script that spits out launch.json with all extension to py files mapping

Wow, ok, I get it now! Thanks for that.

I beleive you’re referring to @ benjavides and not me :slight_smile: I’m just a humble topic creator :slight_smile:

First thanks for all the testing! To answer your questions:

  1. I left several commented out lines of codes in the project because it was working without them but knew would be good suggestions to try different things and figure out a thing to get things working. I couldn’t find a pathMapping that worked but knew that was the key to avoid having to put our project right on the C folder. Regarding the host and port: I used this for vscode to connect to my debugpy server explicitly which worked when launching debugging without process id selection (My guess is that this doesn’t inject the dll that creates the debugpy server and that’s why it connects using the specified info)

  2. This is what I couldn’t figure out. Despite not connecting to my debugpy server (because the errors point that the connection couldn’t been made), why it didn’t work without launching it.

I’ll try later today with your suggested pathMappings.

Regarding your problems with using the “vscode breakpoints” (specified with a red dot in the UI) this is what happened to me when I connected the debugger directly to my server (using the connect section in the launch.json and launching without the pickProcess command).

I will try to increase the debugpy timeout like you said and not launch the debugpy server explicitly to see if it works for me.

Totally agree! It was not a clean solution that’s why I posted it so people like you could take a look and we could get to a cleaner solution together.

Lets not call it a solution but the state I am at currently. I have to check if there is a good way to tell it to add or remove a .py .
Regarding the script file-creation, def a viable solution! But this relies on all files being in the same place. But now we know how and where to do it! :slight_smile:

Whoops. Sorry :slight_smile: Kind stressed and wasted to much time trying to get whats is going on!

But them being commented out is what tripped me off.
I am not sure either but sometimes it would connect directly without issue, sometimes not. But after spending wayy to much time and your commented snippets, I finaly found that they are unrelated.

Again, great work! I would have never looked in to that myself at that level wouldn’t it have been for your pioneering here! You went 90% of the way, I just walked the last 10 :slight_smile: