SRE module mismatch related to PYTHONPATH? (MacOS)

Bugs
1

Hi!

I’m seeing a recurrent error in some macOS environments when running an external bash custom script that creates/activates a Python virtual environment. Once this venv is activated, it installs some pip dependencies or runs custom external Python scripts.

A specific version of Python is used to create the venv (3.12 for example). I can verify that the correct Python version and binary files were used to create the venv. And after activating the venv I can check that the Python version is still the same.

python3 --version
which python3

That outputs, in my case.

Python 3.12.3
/opt/homebrew/bin/python3

The main issue is that even with venv activated with the correct Python version when running pip I get

Error in sitecustomize; set PYTHONVERBOSE for traceback:
AssertionError: SRE module mismatch
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/Applications/TouchDesigner.app/Contents/Frameworks/Python.framework/Versions/3.11/lib/python3.11/venv/__init__.py", line 7, in <module>
    import logging
  File "/Applications/TouchDesigner.app/Contents/Frameworks/Python.framework/Versions/3.11/lib/python3.11/logging/__init__.py", line 26, in <module>
    import sys, os, time, io, re, traceback, warnings, weakref, collections.abc
  File "/Applications/TouchDesigner.app/Contents/Frameworks/Python.framework/Versions/3.11/lib/python3.11/re/__init__.py", line 125, in <module>
    from . import _compiler, _parser
  File "/Applications/TouchDesigner.app/Contents/Frameworks/Python.framework/Versions/3.11/lib/python3.11/re/_compiler.py", line 18, in <module>
    assert _sre.MAGIC == MAGIC, "SRE module mismatch"
           ^^^^^^^^^^^^^^^^^^^
AssertionError: SRE module mismatch

Some other users fall into another set of errors that look like this when trying to create a venv python3 -m venv venv:

Creating Python venv at: /Users/patrickmullady/Documents/StreamDiffusionInstall/StreamDiffusion/venv
Could not import runpy module
Traceback (most recent call last):
  File "/Applications/TouchDesigner.app/Contents/Frameworks/Python.framework/Versions/3.11/lib/python3.11/runpy.py", line 14, in <module>
    import importlib.machinery # importlib first so we can test #15386 via -m
  File "/Applications/TouchDesigner.app/Contents/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/machinery.py", line 15, in <module>
    from ._bootstrap_external import NamespaceLoader
ImportError: cannot import name 'NamespaceLoader' from 'importlib._bootstrap_external' (/Applications/TouchDesigner.app/Contents/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/_bootstrap_external.py)

What these errors have in common is that, somehow, the TD Python framework is getting in the way. I’m not quite sure about what’s happening or how TD manages to do this. One of the users has reported that unset PYTHONPATH fixed the error but I assume it’s not good practice.

So, even though this is a quite abstract description of the issue I think the main point is that something is happening from TD that is interfering with the OS Python environment. Any hints on how to debug or deal with this?

Let me know if there is any other relevant information that might be useful so I can share it here.

Any hints or advice are much appreciated. :slight_smile:

2

Just checking, but from where are you calling the commands? Are you calling them from inside TD or from the OS-Level.

When you are calling them from inside TD and pythonpath env variable does get passed to child processes!

1 Like
3

It’s an external bash script, triggered from inside TD.

(...)
os.system(f"chmod +x {sh_file_path}")
subprocess.Popen(['open', '-a', 'Terminal', sh_file_path], cwd=base_folder)
4

In this case ENV-Variables are still varried over as it is a subprocess.
You can pass a custom environment to the subprocess call.

import subprocess, os
my_env = os.environ.copy()
del my_env["PYTHONPATH"] 
subprocess.Popen(my_command, env=my_env)
5

I was thinking about using os.system(f'open -a Terminal {sh_file_path}') instead of subprocess. Would still the PYTHONPATH override the env variables?