Best practice to import python libraries on a "global level"

Hello hello,
I was wondering what are the best practice/coding workflow to import python libraries on a “global” level?

What I mean is: usually I write my python script in a text DAT or, some sort of execute DAT, and I import the library at the top of the script (for example import datetime).

Now, I’d imagine that every time I run that script, I am importing and re-importing the same library on and on again? If that’s the case, what happens if I have to run some scripts several times every minute? it doesn’t seem very efficient to import the same library hundreds/thousands of times.
Or perhaps TD in the background imports the library only once, the first time the script is run, and that’s it?

On top of this, I would love to know what is the best way to import a python library on a “global level”.
For example I have found myself having several smaller scripts in my network, and in every script I had to import the same exact library.
I was wondering if there’s a way to import a library once (for example using an execute DAT onStart(), to import the libraries once only when I open TD) and then get all the other python scripts to recognise those libraries, without importing them again “locally”?

Any suggestion?
Thank you in advance!!

I recommend to level up your Python-fu by starting to use Extensions - Derivative
Your life will never be the same.

A TD Extension is like a Text DAT on steroids which stays loaded into memory, and all the Extension’s variables, properties, functions & imported packages stay available to all nodes for as long as TD runs.
Then you import python libraries at the top of your extensions, these will then be imported once during extension init (extensions init normally only once when your application starts).

Every COMP can have Extensions, but an easy start is for your application’s main COMP to have a single Extension which does all. Then in the future you’ll probably give each of the larger COMPs in your application their own extension.

2 Likes

Wow,
thank you so much @nettoyeur for pointing me in the right direction!
I appreciate your help.

I must admit that I have found myself opening that wiki page quite few times recently (*), struggled a lot to understand it, found it quite intimidating, and eventually gave up.
I guess is time to face my fears (aka TD occasionally very impenetrable and steep documentation) once for all !!! :grin:

(*) I have been polishing and upgrading some custom components that I use often as drag and drop in my projects, so probably it’s about time for me to look into this properly

To ease your mind, modules only get imported once! After thet the “compiled” version of the module lives in the python environment. Every other import now refferences the same module. This is also a gotcha, as changes to the modules state are also translated to all other refferences to that module!

This also holds true for module_on_demand/importing a textDAT.
The TextDAT will be compiled/run once, and then afterwards stays in memory until
A: The module is changed.
B: The textDAT is run()

This example should show whats going in.


Funny enough, this means that the only real reason to use extensions is to use the binding to the OP itself.

Another interresting thing is that you can create a local and inside a modules base.
Python will now search recursevly inside of all module comps in itself and its parents (up to the toplevel) to find this module. This way you can somewhat create a global “import” (you still have to import, but it is a single source!) for your project.

Thank you so much @nettoyeur and @alphamoonbase for your help.

I am reaching out again because I cannot get the extension to work the way I intend to.

I had a look at the wiki page Extensions - Derivative, (I think I understand about 10% of what’s written in that page) and followed @raganmd excellent tutorial that is mentioned on that wiki page.

However I am still stuck with my problem and cannot get the extension work.

Extensions seem super powerful and I love them (I’m trying my best at least!), but at this point all I am trying to achieve is importing a python library once only (datetime for example), then run some scripts that require that library, without having to reimport the library every time.

I have attached the file I am working on.
On the left is a base COMP with the network I built following Matthew Ragan’s tutorial, on the right (base3) is my attempt to modify that tutorial.
My attempt doesn’t work, I am not sure that I am taking the correct steps or that I am setting up the extension the correct way.

Extensions.toe (8.7 KB)

I have left plenty of comments and took out anything that wasn’t necessary.

I was wondering if anybody would be happy to provide few more details on how to use the extension to import python libraries only once, or perhaps provide a simple working example .toe file for me to have a look?

Thank you in advance!!

Some thoughts for you here @FaustoB.

Like @alphamoonbase pointed out, when you put your import statement above your class definition, the library is only imported once when your extension is compiled. In your extension that might look like this:

import datetime

class GetDateandTime:
...

You can now access that datetime library in your extension without needing to import that again. Moving down your class definition , you can make a few changes like this:

import datetime

class GetDateandTime:

	def __init__(self):
		self.datetime_buffer = op('now')

	def GetDateandTime( self ):

		# this script requires the library datetime.
		# I am trying to import the library once only
		# then run this script without reimporting the library everytime
		
		self.datetime_buffer.clear()
		datetime_object = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
		self.datetime_buffer.write(datetime_object)

		return

Now when you run the command GetDateandTime() you’re not actually re-importing the datetime library, just running the associated python.

If you didn’t want to put this into a DAT, but instead just wanted to get back the date time formatted the way you want, you might do something like this

import datetime

class GetDateandTime:

	def __init__(self):
		self.datetime_buffer = op('now')

	def GetDateandTime( self ):

		# this script requires the library datetime.
		# I am trying to import the library once only
		# then run this script without reimporting the library everytime
		
		self.datetime_buffer.clear()
		datetime_object = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
		self.datetime_buffer.write(datetime_object)

		return
	
	def FetchDateAndTime(self):
		return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

You can now use this directly in a parameter, or in an expression in Touch:

It’s important to note that this only updates when an operator cooks / when the function is run - not a great application in a text TOP - unless you wanted to know when that op last cooked… which this would tell you.

Does that make sense in terms of when this library is imported?

It does! thank you thank you thank you.
It’s 10pm here after a massive day but I had to try this! It works like a charm. I was definitely on the wrong track.
Thank you again @nettoyeur @alphamoonbase @raganmd, all this is of immense help.

@FaustoB, the one other thing to point out here is that you don’t need an extension for this kind of operation. Extensions are very helpful when they belong to a COMP because that COMP has some sense or memory, or has a set of methods and members that need to remain self contained.

For more general functions, a module is also a great way to work in TouchDesigner. Building on what @alphamoonbase pointed out, you can take your same example, and treat it as a module like this:

Here’s that module so it’s easier to read:

import datetime

def UpdateDateTimeBuffer( datetime_buffer ):

	# this script requires the library datetime.
	# I am trying to import the library once only
	# then run this script without reimporting the library everytime
	
	datetime_buffer.clear()
	datetime_object = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
	datetime_buffer.write(datetime_object)

	return

def FetchDateAndTime():
	return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

This does all of the same things, and also only imports the datetime library once. Modules are a really great place to start, and can also be easily accessed by ops all over touch.

These days I usually work with a combination of both modules and extensions. For me, modules tend to be organized as the generalized functions that I might need to access anywhere in touch (your date time use is a perfect example of something that I’d put in a module). For me, extensions belong to component and tend to have some more specific use cases that are focused on extending the functionality of a comp. For example, if I have a data parser that needs to perform a set of transformations on a data set and then push it into a specific data structure.

Very interesting Matthew, thank you for explaining and for pointing out this alternative approach.
I like it, I like it a lot.

A quick questions if you don’t mind.

How does TD know that it has to import that library, if I never run that line of code import datetime ?

If I understood the principle here, that evaluate DAT is just calling FetchDateandTime() and nothing else (I think)?

Great question. This bit isn’t specific to Touch, but is rather a Python behavior / mechanic. The import keyword is used to invoke the python import system. Broadly speaking, in many languages code is written in smaller modules or libraries that are then included in larger projects. This organizes code into smaller chunks that are both reusable and more easily maintained (developing smaller libraries that allow parallel development or management is sometimes referred to as “orthogonal” ).

What is import

import tells Python that there’s a set of functions outside of the default python tool set that are going to be accessed in any given module or library. As a best practice, import statements typically happen at the top of a module, and are outside of any functions so that their scope (your ability to use them) allows them to be used by any function in that module.

For example - this syntax is also completely functional:

def UpdateDateTimeBuffer( datetime_buffer ):
    import datetime
	datetime_buffer.clear()
	datetime_object = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
	datetime_buffer.write(datetime_object)
	return

def FetchDateAndTime():
    import datetime	
    return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

The above is just redundant - if we only import dateteime in the local scope of function, no other function can use it, and we subsequently need to re-import in every function. This is why you typically see this instead:

import datetime
def UpdateDateTimeBuffer( datetime_buffer ):
	datetime_buffer.clear()
	datetime_object = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
	datetime_buffer.write(datetime_object)
	return

def FetchDateAndTime():
    return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

The datetime library is available to all functions, which is exactly what we want. As a final note, you can technically do this:

def UpdateDateTimeBuffer( datetime_buffer ):
	datetime_buffer.clear()
	datetime_object = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
	datetime_buffer.write(datetime_object)
	return

import datetime
def FetchDateAndTime():
    return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

This is just considered bad form since it would be difficult for another developer to find that hidden import statement.

Where does it come from?

To python, the import statement typically refers to a .py file or folder on your computer. datetime, for example can be found in your TouchDesigner installer folder, on my machine thats: C:\Program Files\Derivative\TouchDesigner.2021.15240\bin\Lib

Anytime you use the import statement in python you’re actually issuing an instruction to search the Lib directory for a file or folder that matches the name after import - so in the case of import datetime python is looking for a file called datetime.py in the directory Lib and added the functions in that library to what’s available for Python to use.

TouchDesigner Magic

TouchDesigner extends that same behavior to also include DATs - to really answer your question we’re going to use this trick so we can see a few other pieces at work.

When does it run?!

To see how / when a module runs we’re going to first setup a small test bed:

Here a DAT which contains some helper functions is called myMOD - let’s think of this DAT as if it were datetime. Next we have another DAT that would be where you might author your functions, in this case called exampleMOD.

We’ll use the import statement in exampleMOD to import all of the code from myMOD, which is the same thing that’s happening (nearly) to when we import datetime.

In this example, text2 uses our exampleMOD module. If we run that DAT we’ll see in the text port a cascade of debug statements that will let us trace back to when each piece element loaded and run:

python >>> 
2021-11-12 16:50:07 myMod Init (Debug - DAT:/project1/myMOD fn:<module> line:11)
2021-11-12 16:50:07 hello world (Debug - DAT:/project1/myMOD fn:superFunc line:5)
2021-11-12 16:50:07 hello world 
  (Debug - DAT:/project1/exampleMOD fn:TestFunc line:7)

When we run that module for the first time, that’s when we first load all of the functions from myMOD, next we see that the function superFunc ran, and finally we see that our function in exampleMOD ran. If we run that text2 dat again:

python >>> 
2021-11-12 16:50:07 myMod Init (Debug - DAT:/project1/myMOD fn:<module> line:11)
2021-11-12 16:50:07 hello world (Debug - DAT:/project1/myMOD fn:superFunc line:5)
2021-11-12 16:50:07 hello world 
  (Debug - DAT:/project1/exampleMOD fn:TestFunc line:7)
python >>> 
2021-11-12 16:52:04 hello world (Debug - DAT:/project1/myMOD fn:superFunc line:5)
2021-11-12 16:52:04 hello world 
  (Debug - DAT:/project1/exampleMOD fn:TestFunc line:7)
python >>> 

We only see the function calls, and not the import - now that our module has been imported it will be persistent through this session with TouchDesigner. At this point all of the functions in our myMOD module are now available without reimporting them, even if we add another text DAT, and run the function there:

TL;DR

The quick answer is that import is a keyword in a module, and Python will automatically run a helper function to load externals when it sees this keyword the first time a function in that module is executed.

import-example.toe (3.7 KB)

https://docs.python.org/3/reference/import.html

Thank you so much for the detailed and clear answer!
This is exactly what I was after.

I understand now the behind the scenes mechanics, your example with DATs is very clear.
Modules are probably the best approach for my specific scenario, but I’ve learnt heaps about extensions as well, they both are very powerful tools.

Thank you again @raganmd @nettoyeur and @alphamoonbase for your help.
Thank you more broadly for your excellent posts & responses here in the forum, and for the quality components & tutorials that you generously share.