TouchDesigner's Python distribution missing include/libs directory required for runtime torch extensions

General TouchDesigner Discussion
1

Am working on getting an old version of stylegan working inside of td, which requires some ad-hoc torch extensions be compiled via torch.utils.cpp_extension, which looks up a variety of python headers and tries to link to python311.lib as part of the build process.

Historically, I’ve been able to just use the same 3.11.x version of Python installed separately to create venvs and avoided interacting directly with td’s distributed version, but the torch utils get very grumpy if things are compiled with even different patch versions.

I was able to hack this together because the headers and dlls are distributed in Samples\CPlusPlus\3rdParty\Python, but this is totally wobbly and took me a bit to figure out. Would it be possible to include them in the build in their “normal” locations? This seems to already be the case on macOS where everything is bundled under Python.framework.

Thanks!

2

Hi, just to be a bit clearer, what do you expect to be the correct location? and do you have a link to the Pytorch doc for those ?

4

Looking at regular Python installation on Windows, those headers are located in
C:/Python311/include
next to the Lib and python.exe

The problem with that with our installer, that would mean they would end up in
C:/Program Files/Derivative/TouchDesigner/bin/include
Which isn’t a great spot for headers for a specific library. At the very least it would need to be include/Python. I think where they are right now makes the most sense, since that is relative to the sample projects, and we can add other headers there under 3rdParty in their own subdirectories.

If you have other suggested locations though, I’m happy to consider them.

6

what do you expect to be the correct location

I’m not totally familiar with how Python is typically packaged, but from looking at my global python 3.11 installation, a conda managed 3.7 version, and the macos td distribution’s python, all have lib/ and include/ as siblings of the python executable.

and do you have a link to the Pytorch doc for those ?

https://pytorch.org/docs/stable/cpp_extension.html

which seems like it just calls into

https://setuptools.pypa.io/en/latest/userguide/ext_modules.html#extension-api-reference

I had to use extra_include_flags and extra_ldflags pointing at the samples directory in order to get things to work.

Without those, you can see it’s looking for C:\Program Files\Derivative\TouchDesigner\bin\Include.

RuntimeError: Error building extension 'bias_act_plugin': [1/3] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\nvcc --generate-dependencies-with-compile --dependency-output bias_act.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Users\char\Documents\GitHub\stylegan2-ada-pytorch\.venv\Lib\site-packages\torch\include -IC:\Users\char\Documents\GitHub\stylegan2-ada-pytorch\.venv\Lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\char\Documents\GitHub\stylegan2-ada-pytorch\.venv\Lib\site-packages\torch\include\TH -IC:\Users\char\Documents\GitHub\stylegan2-ada-pytorch\.venv\Lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include" "-IC:\Program Files\Derivative\TouchDesigner\bin\Include" -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_89,code=sm_89 -std=c++17 --use_fast_math --allow-unsupported-compiler --use-local-env -D_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH=1 -c C:\Users\char\Documents\GitHub\stylegan2-ada-pytorch\torch_extensions\bias_act_plugin\266338fdc85262ebdea23fddc9399ca6\bias_act.cu -o bias_act.cuda.o

And trying to link to C:\Program Files\Derivative\TouchDesigner\bin\libs

[3/3] "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.43.34808\bin\Hostx64\x64/link.exe" bias_act.o bias_act.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:C:\Users\char\Documents\GitHub\stylegan2-ada-pytorch\.venv\Lib\site-packages\torch\lib torch_python.lib "/LIBPATH:C:\Program Files\Derivative\TouchDesigner\bin\libs" "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64" cudart.lib /out:bias_act_plugin.pyd
FAILED: bias_act_plugin.pyd
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.43.34808\bin\Hostx64\x64/link.exe" bias_act.o bias_act.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:C:\Users\char\Documents\GitHub\stylegan2-ada-pytorch\.venv\Lib\site-packages\torch\lib torch_python.lib "/LIBPATH:C:\Program Files\Derivative\TouchDesigner\bin\libs" "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64" cudart.lib /out:bias_act_plugin.pyd
LINK : fatal error LNK1104: cannot open file 'python311.lib'
7

I’ve figure this out so I’m not particularly pressed at the moment, but it was a bit challenging to track down do to other downstream failures it caused. It does feel a bit strange to be referencing stuff included in the Samples/, but mostly wanted to post this in the event there was an easy way to include these in a standard location and in case anyone else runs into the same issue in the future.

8

OK thanks for the feedback. I’ll consider it a bit more.

1 Like