RFE: integrate Poetry package manager with TD

Wishlist and RFEs
#1

@Ivan here’s a wild idea:

Since I discovered it a year ago, I only use Poetry as package manager for my python projects. It’s amazingly simple, beautiful, and easy and slick as you wish pip would be. It’s so well designed it brings a tear to my eye :wink:

just read this: https://python-poetry.org/docs/basic-usage/
and https://python-poetry.org/docs/cli/

The cool thing is for each project you create with Poetry it creates a local environment which you can easily commit to git update/change/send to a client. By default Poetry uses a venv it fires up for each project, but that will not work with TD and we also don’t need the venv feature, we just need Poetry to manage packages and that TD knows about them being installed locally.

I currently already use poetry with TD projects, but it requires some hacks for each DAT to be able to import these locally installed packages which makes it all a bit cumbersome to use.

My dream is that TD would ship with poetry included, and that I for instance would only have to type
poetry add pycuda in the textport to include the pycuda python package with my current .toe project.
That would mean from then on I can type
include pycuda
in all my TouchDesigner DATs and it would just work. (just like the Requests library that ships with TD)
Also that would mean a new directory Lib/site-packages would be created next to my .toe file where in this example the pycuda package (and its dependencies) will be installed to (so they are not system-wide installed but per project instead).

poetry does all sorts of amazing things in the background, like create project.toml and .lock files which you can commit to Git so it’s guaranteed any client / coworker opening a .toe project 6 months from now would get those exact same package versions I specified when delivering the project.

thanks for reading all this and try to take look at poetry if you have some spare time !
cheers idz

6 Likes
Popular Python Modules
Realtime Speech Recognition using Python
#2

+1 for a package manager for TD

1 Like
#3

I have a working prototype of a poetry COMP. It creates a local lib/site-packages and installs all your (resolved) python and tox dependencies . And you can create python packages holding tox files, publish those to pypi … works nicely, but I definitely prefer a native solution and therefore fully support that RFE.

1 Like
#4

My PIP-Implementation (windows only) also creates a library-folder in the project directory and adds the refferences on startup. So, in theory you can also just put the libs in the Lib folder of the project.
Also, the Import_Library method will come in handy as it will download the module from PIP and install if it does not exist and returns the module afterwards (or directly if it already exists)

Nothing against the RFE, def nice to have tight integration.

1 Like
#5

Thanks for the ideas and the push. I’ve posted this in our internal RFEs. I agree this is important.

2 Likes
#6

imho this should not just use poetry for managing python packages, but to also handle tox files as python packages. This way you automatically get proper version and dependency management for tox files. And you can use pypi or any custom hosted python repository as your tox repo

Another interesting opportunity that would arise if using poetry as a built in package manager is to use the “necessary” pyproject.toml to configure the toe file. Stuff like project.paths or any custom variables could all be group in that one

1 Like
#7

I’m going to look into this. Anyone that wants to share some simple example setups or specific features to look at, please post them here.

#8

Hi @Ivan

here is a repo showing the main usecases https://github.com/AchimKern/poetry-managed-td

Hope it runs without errors on your machine. If not or if you have any questions please let me know

3 Likes
#9

Another feature: any project-specific packages which one defines using Poetry should always take preference over any system-wide installed package, or a package shipped with TD. This means it would be trivial for a user to override built-in NumPy with a newer/older version using Poetry. This is just a matter of ordering the include paths I think.

3 Likes