Intermittent laser glitch

I recently acquired a Laserworld DS2000RGB and am spending some time getting my head around the nuances and limitations of its display abilities. At the moment, I’m experiencing an intermittent badness in developing some test patterns.

Notice to newcomers - If you’re experiencing this issue as well, please post your system information in this thread in the Bugs category.

Symptom

Most of the time, things are fine, but occasionally the laser completely freaks out, then shuts off light for several seconds before becoming responsive again.
In words: The servos go absolutely wild, drawing random patterns across the scan range. The RGB modulation also looks like it bugs out randomly, but mostly the trace stays shades of white.
Here’s a video showing a simple test pattern, which has a small glitch (misplaced line) a few seconds before the misbehaviour https://www.youtube.com/shorts/jMvvuJFJ9W4
Once this occurs, the laser remains off for a few seconds (5+) and eventually switches back on, showing the expected pattern.

Occurence

  • I have not observed this behaviour when using the proprietary ShowNet admin tool to display test patterns.
  • This occurs in the network view, whether or not I’m messing about with the contents, although I suspect it’s less frequent when it’s running ‘hands off’ without me messing around with the network.
  • I tried reducing the frame rate to 40fps but this did not resolve the issue. In fact, the behaviour occurs even at times when the FPS is exactly on target; furthermore, if I add hog CHOPs until the framerate tanks, the laser continues to draw
  • Sometimes (perhaps more concerning) the laser device CHOP raises a warning ‘Failed to open device’ after this occurs, and I need to restart TD in order to re-establish control over the device.

Setup

The laser is being controlled from my laptop which is connected directly via ethernet cable. I am unsure whether it’s streaming ILDA to the interface or whether the laser device CHOP has built-in compatibility with the proprietary ShowNet protocol (and I would very much like to dig deeper into how the laser ops work so I can get the best performance from this device; the documentation I’ve found so far is not especially detailed).

Anyone else had a similar experience, ideas about the cause or further tests?

Bonus: Here’s what the badness looks like. Kinda cool, were it not a significant problem.

Hey @groundhogstate,

thanks for sharing this detailed description, could you additionally share with us your test file so we can try to replicate the behavior here?
Could you also list your computer’s hardware and OS as well as TouchDesigner version you are currently using for this?

cheers
Markus

Here’s the file, and system information. TD version 2023.11290.
Screenshot 2024-03-19 150725

laser_interface.toe (8.9 KB)

1 Like

Hi @groundhogstate,

thank you for sending this, we are looking at the issue.

cheers
Markus

Thanks, how’re you going with it? Did you manage to reproduce the behaviour?

I looked at the ethernet packets being sent by using wireshark. In the case of the ShowNet admin tool, there are some occasional re-registrations, but otherwise things run smoothly. When using TD, the packet history shows more frequent re-registrations, and then some anomalous behaviour which I think, but haven’t timed precisely, coincides with the main features of the symptom. Some screenshots below in case they are helpful.

Would there be any other tests you would recommend in the meantime?

Screenshots

Running ShowNew admin tool

There is a handful of these strewn across a couple minutes of uptime

Normal running looks like this, which is pretty similar to the normal operation of the TD interface.

There are a couple of places where someone drops out

But no other suspicious behaviour after several mins of runtime.

Using TouchDesigner

On first connection, the exchange is more or less the same as when using the admin software.
This is followed by a few other membership reports or registrations as well; they could more frequent though. Eventually it gives up, and there are some anomalous packets; in particular, they are just about a carbon copy to the packets exchanged when the laser is switched on. NB I ran this test first, chronologically. I’ve sent some further screenshots to your inbox @snaut

We haven’t been able to reproduce the issue yet but we have reached out to LaserWorld to get their help with reproducing/solving. I do at least recommend updating to the latest TouchDesigner official build which uses the most recent ShowNET SDK version (1.3). 2023.11290 is still on an older version, although I can’t reproduce in either build so it’s unlikely to be the fix.

That’s interesting to know about your differences in packet activity between TouchDesigner and the ShowNET admin tool. We don’t have direct control over that aspect since it’s all handled by the SDK, however it may point to a misconfiguration on our end. We’ll continue to look into it.

Heya, thanks for the tips.

I’ve upgraded to the latest TD and in fact due to another issue performed an OS reset, which hopefully eliminated a lot of variables. However the bug still occurs. Somewhat worryingly, I did start up with a fresh file on a friend’s Macbook (as above, I’m using Windows) - the error syndrome still occurs. At this point I’m concerned that the issue could be in the laser interface itself, but, as I noted above, the problem has not arisen when using the ShowNET admin tool.

Just to be sure, are there any other configuration details that might have been overlooked? For example, I could only establish a connection via ShowNET with my laptop and the laser set to static IP configuration; LaserWorld also warn about possible issues with WiFi being concurrently on, but I’ve got that set up so it uses a different address space to the laser. Anything different on your end or that you’d recommend fiddling with?

I haven’t purchased a license for other LaserWorld software so that’s not feasible test immediately, but I think it could be worth trying other software in the interim.

Oh another thing that might be relevant, @eric.b / @snaut - the laser in question is actually a DS2000 Mk3. If you’re using another Laserworld product to test, perhaps you have a newer generation? I have no idea whether their interface design changed between mk3 and mk4, but it might be a consideration.

A potential breakthrough: I’ve heard from another user who had a similar issue while using an Etherdream interface. However, in their case, the error state occurred using windows, but not a Macbook; It was alleviated after switching from an AMD 5950x to an intel 14900k. Strangely enough, as earlier in the thread, I’m using an Intel.

Further, I’ve since tested using a Mk4 and the situation is unchanged.

I’m testing with a DS-1000RGB MK3 so there doesn’t seem to be an issue between MK3 and MK4.

If another user experiences the same issue using EtherDream with the ShowNET laser then that means it’s unlikely to be a problem with the ShowNET control API. It could be that the points generated with the Laser CHOP are causing the problem – do you only experience the issue with a specific Laser CHOP input, or others as well? Does lowering the output sample rate have any impact on reproduction frequency?

The error syndrome doesn’t care what pattern I’m tracing. It sometimes even happens when the laser is on but there is no output. That is, I’ll have the laser device CHOP inactive, but the device on, so no light; and then it’ll just do the bad thing and return to the off state (with a small amount of dead time in case I do reactivate).

I looked into the sample rate; it was originally much higher than the DS2000 would deem sensible (48 kpps, which I think is the default for the laser device CHOP). I found through the Shownet admin tool that the on-board sample rate for reading from the SD card is 20 kpps. So I dropped the output from the laser CHOP to match, and it seems like the laser is better behaved but not totally resolved, which is interesting. That is, the glitch happens, but a couple of times per hour rather than once every few minutes. Which doesn’t seem to get drastically better if I drop the sample rate to 18 kpps; I can try with an even lower setting like 10 kpps and see if it’s improved further.

Why do you think the sample rate could be linked to this? Does it change the rate at which TD sends packets over ethernet to the interface?

The rate of the Laser Device CHOP sets the scan speed of the laser via the API, so it may be that the error is less likely to appear at lower scan speeds.

That is strange about it glitching even when the Laser Device CHOP is inactive. When active is toggled off we release the device handle in the API and clean-up, so we no longer have control over the laser device at that point. That now makes me wonder if the issue is with the laser itself, as I also still haven’t been able to reproduce with my test laser.

Thanks for persisting with this one, @eric.b.

Considering that implementation detail, I might be mistaken with that remark about the CHOP being inactive. I’ve been doing a lot of installation development the last week or so and making observations on the way, rather than rigorously testing for the moment. So it could be that I just had no input to the laser device as opposed to it being inactive; this is something I’ll factor in to future tests.

I did have some correspondence with LaserWorld support and have been told ‘If [a] switch or router fragments the data packets in order to be able to send them faster, a similar error can occur’; to my (admittedly non-expert) mind this seems like a hint in the same direction as the observation about sample rates.

It’s totally possible that it’s a hardware issue, although it’s occurred with two separate units on my end.

I’ll be focusing mostly on laser programming development for the next week as I’ve got a show to prepare for, so I’ll let you know if I find anything else interesting or a hotfix. Will return to more systematic testing after the fact.

Cheers

It seems I’ve had similar issues with Helios DAC, Etherdream, and my own custom LaserCube python script

For me it happened when opening and exiting a node sometimes
Laser goes wild no matter the X and Y size limits and same for RGB intensity and this is a BIG NO NO for some use-cases safety speaking.

I’v had (almost) no issues on my last shows using my python script for laser cubes. I’ve yet to write a custom C++ OP for the laser cube to see how it goes

Will try to reproduce them and post wireshark logs as well

Thanks for the input @Dorian. “Big no no” indeed. When you say python script, is that running in a TD project or do you have a standalone implementation? Can you provide some more details about how that’s built?

I have had Wireshark running during my development lately and have observed that the misbehaviour coincides pretty consistently with some sort of anomalous wireshark activity (I haven’t been bothered to try and time it down below a second’s accuracy, but there’s a striking correlation).

The exact details seem to vary a bit but some possibly high-value instances are just below. I say ‘possibly’ because this is beyond my network-fu.


I have a hypothesis that this could be mitigated by running the laser in a ‘pulsed mode’. (edit: This has not borne out; the pulsed-mode operation does not prevent the issue, but it does only occur when the laser is active.) This seems to occur when the laser device chop is active; I wonder whether there’s just ‘too much traffic’ and that deactivating the chop for a short period intermittently might give enough time to purge some buffers or backlogs. If indeed that’s how ethernet actually functions.

1 Like

Oef… i have the same exact behaviour on my end. Including the catastrophic glitch knocking out the control for a couple of seconds (10 seconds). And sometimes needing complete restart to refunction.

Found any updates on this?

1 Like

Welcome to the party, @Wesb!

No helpful updates from me. However, I regret to inform the world that in my testing last night, I witnessed a bunch of the aforementioned failure mode which exhibited no such cascade of errors as indicated in my above posts about Wireshark… So that technique is not a guaranteed lead.

Soon I will try a different setup, using a router as an intermediary to allow using DHCP. I note that I’ve failed to find the device when using auto IP assignment, but static IP at least allows the connection to establish. Edit: This was not successful

See also this other documentation which I’m putting here as a bookmark.

FYI @eric.b @snaut I am 99% confident I have isolated the bug to the Laser CHOP.

Edit: Or… I was. see below. Things were learned.

I have created a simple network, below, which produces output with two major differences.

  1. I have yet to observe the major failure mode described in OP
  2. There is a more subtle glitch which has also disappeared. As seen in this video, there are ‘flickers’ where the laser path deviates from the intended path.

I had wondered whether these were connected; apparently they are, and the culprit is somewhere in the Laser CHOP’s implementation - not the device interface. Aside, this has tended to happen much more frequently than the catastrophic failure, and the absence of this specific mode is what gets me to 99% despite less observation time so far.

To be sure, I plugged all of the outputs from the Laser CHOP’s operator snippets into the device, and they reliably produced the error, i.e. it’s independent of whether the laser CHOP is fed a SOP or CHOP input.

On one hand this is a relief; as mentioned, this alleviates a safety issue (and a risk to any production even if the projection area is uninhabited). On the other hand, the rendering of basically anything is poor and so quite a lot of workaround will be necessary in lieu of a bugfix in the Laser CHOP.

Heads up to @Wesb and @Dorian. If I develop any useful techniques to work around the unavailable laser CHOP, I’ll send them your way.

Network diagram:


It is slightly more complex than minimum-viable because I wanted to make it interesting to look at.

Edit: Oh, hot dang. Literally five minutes after I post this, the major failure occurs again. It seems that excising the laser CHOP is more reliable but not actually a solution.

Further checks:

  • Loaded my project with hog CHOPs. Didn’t trigger.
  • Opened all the expensive programs I could and drove my CPU utilization to 100%. Didn’t trigger, although, did freeze the frame.
2 Likes

Hey there,
just wanted to let you know that I have to witness the same issue with my DS1000 MK4. I am far from being a TD professional, so can’t really add some useful hints or hypotheses to your elaborate investigation so far… but wanted to let you know about this specific model which hasn’t been mentioned yet.
I am feeding a rather simple SOP into Laser Chop and it shows the same, erratic malfunctioning as described by all of you.
Best
Bertrand

1 Like

Hi There
Im Experiencing the same problems. Im using a Laserworld DS1000 MK3 as well as an Shownet to Ilda interface. I observed that the frequency of the Bug appearing is somehow linked to the System that Td is running on. If i run it on my Laptop, an generic Thinkpad the glitch appears far more frequent opposed running it on a moore beefy media server. It still happens sometimes. Perhaps this is the reason why you guys from Derivative couldn’t reproduce the error im assuming you are using decent enough hardware wich is probably optimised for Td.
In this state the installation of lasers controlled by Td needs an optical barrier to ensure a proper safety zone. I also found a Workaround in my case by using multiple devices and multible laser chops that i feed from my sobs. The glitch does not happen simultaneously therefore at least one device always has an output. This wont suffice for more demanding installations but whatever helps i guess so im putting it out there.
Cheers

1 Like