Maintaining a persistent hardware connection with a python object

Beginners
1

Hello all,
I am not new to TD, but I am new to extensions. I thought it would be a great learning project to create my own component that manages the hardware connection of the Canable CAN interface and uses the Python-CAN library.

So far I have:

  • Installed Python 3.7.7 (python-3.7.7-amd64-webinstall.exe)
  • Installed Python-CAN (pip install python-can)
  • Created my own custom component & Made my own extension for that component.

I am now trying to create one PulseButton that initiates the connection, then different functions for checking connection status and sending and receiving.

Short Version of the problem:
When I successfully create the connection, there is a Python object returned that represents the connection (with methods that I can call on that connection like status). But when I then create a different function to check the status, it doesnt know anything about the connection… “UnboundLocalError: local variable ‘bus’ referenced before assignment”
How do I maintain this Python Object “globally” to reference in different functions?

Supporting Info:
This successfully connects and returns a python object representing the connection:
bus = can.interface.Bus(bustype=‘slcan’, channel=‘COM5’, bitrate=250000)

I can then immediately do this:
print(bus.state)

Which prints this:
BusState.ACTIVE

A few things I have tried based on forum research and previous suggestions:
I tried the suggestion of some of the developers from old forum post threads that suggested storage:

vcan.store(‘busobject’, bus)

bus=vcan.fetch(‘busobject’)

…but the problem here is that when I run bus.state it shows the status at the time of the store (so if I unplug the USB port, it still shows Active).

I also tried modifying “slcan.py” to use
class slcanBus(ThreadSafeBus):
…instead of the usual…
class slcanBus(BusABC):
…but that didnt help either.
https://python-can.readthedocs.io/en/master/bus.html#thread-safe-bus

Any suggestions would be appreciated…I am starting to wonder if I should try to use the serial CHOP somehow…presumably because it would maintain the connection state for me…not sure if that is the right way to go.

My current TOE file is attached…I have tried lots of different things.
vCan.6.toe (6.1 KB)
Thx!
Steve French (aka frenchy) (aka VoltVisionFrenchy)

2

Hey Steve! Hope you’re well -

I’m not familiar with that library in particular, but two things that I’m like 95% sure are keeping this from working as you expect:

Creating the bus object, I assume initializes and generates it’s attributes, like bus.state, but there’s nothing updating that state parameter after initialization. So even if you could print the state twice, and unplug between, I think you’d always see the state stay the same. I think it’d be great to hear from some other python wiz’s though because I’m not 100% sure how python modules handle “listening” to things like this over time -

Though as far as TD is concerned, I think it would block the thread, and your interface would appear to freeze indefinitely or until time out.

Other than that, I think the .store and .fetch methods would do what you’d expect, you can also create a variable in the extension during initialization, self.A = None, then update that with the bus object when initialized. To have this persist across saves, and initialization, you can use the dependable storage in td, but regardless I think you’ll have the same issue.

One solution I can think of is to save the object in storage one of those ways, and use a serial DAT with callbacks attached to the same com port to push certain changes to the stored bus object… but that might open a can of worms, depending on how much I/O the device has that changes. Not sure that would catch when the device goes inactive either.

You could also take a dif approach, where a timer CHOP for ex runs a 1 sec loop, and at the end of each cycle, pings the serial DAT, and based on what it receives or doesn’t, updates the stored object.

A bit more of a brute force approach, but the timer chop would allow you to “check” stuff on a loop, with out blocking the thread.

1 Like
3

I agree with Lucas. Best way is to store your connection in an extension. If you put it in a property (you can do a Python search if you’re not familiar) you could query the object whenever something tries to access it.

This will never give realtime updates to TD when the object is disconnected. The only way to do that is to do the occasional automatic query as Lucas suggests.

1 Like
4

Ooh python search? that one is new to me, how’s that work?

5

Haha don’t know if you’re kidding Lucas, but I just meant a search for Python properties on the internet

6

OH @Ivan google search - ya I got ya haha, I thought you were referencing some way to search a python extension or something like getattr etc. My bad!

7

@lucasm yeah I was like “I’m pretty sure Lucas knows how to look up Python documentation.” :joy:

1 Like
8

@lucasm @Ivan,
Thank you both very, very much for your responses! I am definitely not a Python programmer (yet) so it is awesome to see how you guys would handle it. I respect you both very much.

I did successfully store the bus object as a property of the extension and I am now able to successfully send and receive from TD!!! I am struggling now trying to keep up with my incoming 100ms status messages. I am polling them, but I am always behind because of some kind of message buffer. Do either of you know of any TouchDesigner tutorial/examples that handle “buffer and notifier” like this answer? (they also handled storing the persistent bus connection differently)…I am working to port this code into TD now. (I have both the PCAN and the Canable interfaces, so ideally I will be able to make a component for each…lots more examples with the PCAN).

@Lucas, sorry for being dense, but when calling code based on an interval (1sec or maybe 50ms), why is the “Script CHOP” method better than the TimerCHOP/Callback method? The TimerCHOP method just intuitively makes sense for me, but maybe I just dont understand the “Script CHOP” yet.

Thanks and talk soon!
-frenchy

9

Whoops! Good catch, I actually meant to say timer chop as that was the node floating around in my head. Good catch! Def the way to go

10

I’m not familiar with CAN, but the example you linked on Slack Overflow looks like it should all work in TD if you have the libraries working. The whole buffering thing looks to be taken care of within CAN. If you have specific sticking points post them here with error messages.

11

Ivan,
Thanks again for your help!!! No errors, but I have I have 3 very short specific questions regarding performance optimization (I know this message seems long, but most of it is optional supporting data and thoughts that you can ignore):

My most recent Hardware component TOX is attached.vCAN_v2.tox (3.2 KB)

Short version of the questions:

Any obvious suggestions to help me increase my framerate regarding these things?
Thing1: Best method to read incoming data from the hardware interface?

  • TimerChop @ 25ms (laggy results below) or ExecuteDAT @ each frame or something else? (the executeDAT seems to perform way better)

Thing2: How should the hardware component pass incoming messages to the rest of the application to consume?

  • SerialCHOP uses a table, but for me writing to ComponentParameters was much better (results below)

Thing3: Wouldn’t a callback system be way better than polling?

Below is optional Supporting Info:

Results I got using the ExecuteDAT (onFrameStart) and Table Method:

  • Framerate was 14-15fps and more laggy response than the ComponentParameter method
  • Summary: writing the incoming data to ComponentParameters seems more performant than writing to a table

Results I got using the TimerChop and ComponentParameter method:

  • 60fps @100ms Interval (beep is great) LAGGY RESPONSE because not keeping up with messages!!!
  • 60fps @80ms Interval (beep is great) LAGGY RESPONSE because not keeping up with messages!!!
  • 60fps @60ms Interval (beep is great) LAGGY RESPONSE because not keeping up with messages!!!
  • 40-50fps sometimes 5fps @50ms Interval (beep is sometimes noticeably laggy) sometimes laggy RESPONSE!!!
  • Summary: In general the TimerCHOP is not as performant as the ExecuteDAT…I am going back to the ExecuteDAT “onFrameStart”.

Acknowledging the SerialCHOP Data Interface method:
I have never used the SeriaCHOP before, but I checked it out and notice that its output is a 10-row table that acts as a software buffer for the last 10 received values. This allows the user to access the data as a table. The serial messages are NOT stored as parameters…it is interesting that I found the parameter method more performant.

Application Info (optional):

By the way, if we zoom out a little bit, here is what I am actually trying to accomplish…

Goal:

  • To create a Test Platform UI that interacts intimately with a multi-node Robotic System

Given:

  • 3 independent pieces of hardware talking on the same CAN bus (electrically similar to DMX lighting, but multi-master, hardware CRC and delay-free hardware arbitration)
  • Hardware1 sends out 1 message (for now) at the rate of 100ms
  • Hardware2 sends out 1 message (for now) at the rate of 100ms
  • Hardware3 is TouchDesigner (Test Platform UI)
  • More messages are expected to be required on the bus, so improving fps performance is ideal.

Processing messages:

  • Write incoming messages to file (would be nice to turn writing on/off during certain events)…have not implemented this yet.
  • Display “real-time statuses” on UI (60fps? less?)

thx!
-frenchy

12

I didn’t realize you’d have such a high throughput of data, perhaps the serial DAT call back could be utilized to push only the relevant message when they come in, to the appropriate variable in the python object, or if you are able to push straight to custom parameter that’s even better.

Is the python “bus” object necessary? what role does it play in the translation of the raw serial, to what TD has to work with?

If there’s enough info on the protocol used over serial, it may be more efficient and simpler to just create logic in the serial callback that “reacts” to messages that fit a certain criteria - and writes the data to where it needs to go.

You could maybe simplify things some more too, by creating a lookup table DAT, where column 0 is the raw message identifier, then column 1+ contains info like the Parameter name to send the data to, or the function to use to filter / modify that raw data etc.

Also a good idea to stick all this into a base or container COMP, and use relative paths, push the data up to the parent component, keeps it nice and modular that way.

1 Like
13

Lucas,
Thanks for the response! I looked into the serialDAT and I do not think that you can have more than one connection to the same serial port…since the Python-CAN library is already connected to the serial port, the serialDAT cannot connect (I think). See this screenshot that shows the error…

Warning_SerialCouldNotConnectoToDevice
Warning_SerialCouldNotConnectoToDevice813×625 61.1 KB

I confirmed this by Disconnecting the original, then the SerialDAT could connect (error went away)…but now the original could not connect until I disconnected the SerialDAT.

In response to your question about whether the python “bus” object was necessary…my main entry-point into all of this was at the bottom of this Canable page here. The Getting-started for Python shows the very first thing being:

bus = can.interface.Bus(bustype=‘slcan’, channel=‘COM0’, bitrate=500000)…
…which for me is COM5 and 250000kbps bitrate.

Perhaps you are right that accessing the SerialPort “more directly” would be faster. I am newish to Python and willing to learn, but abandoning the Python-CAN library seems like a risky decision. It seems very “rich” including built in ways to do logging (which I would LOVE to do next)…and error handling… and listening… and notifying…but if I am “attaching it to TD”, then do I sacrifice or re-build this functionality in TD? It is like attaching two aliens together for me…who has the better leg?..do I really want to build a new leg? The library seems professionally done and I am NOT a professional Python developer!! So the more I leverage the better.

Thanks again! I am wrapping my head around your last two comments now.
-frenchy

14

Innteresting, so it sounds like the library initializes the bus object, fills it with some relevant data from the serial connection, and does in fact maintain the serial connection?

The main problem is that I don’t see a way for the python object to automatically receive data with out stalling the TD thread by checking continuously… which is not a great solution anyways as discovered.

Seems like it’s complex enough to warrant not recreating a whole bunch of functionality in TD as you said as well.

Looking through the docs some more, I see this about listeners:
https://python-can.readthedocs.io/en/master/listeners.html
Which might imply callbacks and the ability to react to data when received instead of polling etc.

You could potentially run a separate python script in windows that uses this functionality, and have that separate python script pass data into TD in real-time when it gets it.

The benefit of this, is you can get started with more or less copy pasting their demo / getting started code, and then once you can confirm it’s spitting out data you want with print statements, then work on passing that into TD. It also works on a different windows thread so won’t lag down TD.

Passing stuff into TD from a running python script is a totally dif topic, but not too hard. Could send it via UDP / localhost, and leverage the UDP in dat with callbacks. and use python there to route the incoming data where you want to custom params and such.

1 Like
15

Lucas’ suggestions of reacting or polling in a different thread seem good. This kind of stuff can be very different depending on the library, and to be honest is not my area of expertise.

At this point I am watching this thread to see what you guys come up with, but not sure I can be of any help for best methods!