I am currently working on an UnrealProject that contains interactions with TouchDesigner. We are currently using a sort of Spout passthrough, where a render target is sent as an input to a TOX that simply passes it to a spoutout. On the otherside TD was sending a spout texture to UE receiving it via the tox.
As seen on a TD forum post a while ago, there seems to be some sort of problems regarding window focus and spout that I cannot manage to fix. It also seems, according to certain people, to be a sort of tick sync issue between the two software instances. The problem doesn’t seem to go away in a packaged game either.
So i was wondering if anyone had any recommendation of the best way to simply pass a video feed from a render target to touchdesigner and vice versa, with as little delay as possible and without this significant FPS hit from syncing issues. We originally used spout because we assumed it would be the most cost efficient solution. Tried NDI too with any possible buffer, cook timeout and tick settings, without any success into reducing latency and fps drops.
My own experiences weren’t with TouchEngine, but the principal tweak for stabilizing performance when working with UE and TD on the same system that I had to do was setting UE to a fixed frame rate that matched TD. UE defaults to a non-fixed as fast as possible rendering strategy which would get throttled by TD running fixed on the same host. This would generally address the ‘window focus’ issues for me but, it’s been a while.
Hi @Lainhehe - could you describe the actual problem you’re seeing? You mention some reported problems other people have run into, but don’t say exactly what you’re seeing in your case.
It’s also not clear exactly what the sequence is - am I correct in understanding you’re doing UE texture → TouchEngine-UE → Spout → TouchDesigner (and your own processing) → Spout → TouchEngine UE → UE texture, and TouchEngine-UE is only acting as an interface to Spout?
If so, why involve TouchDesigner? Can’t you do all the work in TouchEngine-UE?
TouchEngine is basically a simple passthrough. I involve TouchDesigner because the project is bigger than that, and TD is the last step of the content we’re producing and includes post processing etc. The TOX has three major functions, while the chop one is probably not really an issue. although it has some delay too. I have an OSC In distributing data to control certain aspect of the UE project, a spout out, that takes two render targets and sends them to Spout/Ndi to catch them back in TD (Main feed passthrough), and the opposite, a Spout In/NdiIn that takes a texture for a specific mesh into unreal. This texture is managed on TD.
The problem ive seemed to run into, which I think @flowb understood well, is related to window focusing and some sort of tick syncing with spout between the two instances of softwares. I remember trying to limit fps and it worked sometimes and less other times. I am not closed to the idea of changing spout for something else, as right now Im reexperimenting with NDI to see if it works better but I assumed spout would be the most optimized usecase for this project as its directly shared. More or less, the problem is just abusive delays and not so good fps. 95% sure it is not performance related. here is a screen of the tox
after rechecking, I already fixed the framerate to 60 with no success on UE. also wouldnt that mean that any fps variation on TD side would create this throttle anyway? going lower than 60 is not really a viable option. I remember a day where ive set TD to 30 and it was a lot better while also being very far from perfect
Just to be clear, I don’t think requesting 60fps in UE fixes all sync issues. However, allowing UE to run at whatever frame rate it chooses has caused the focus issues for me in the past, so when I’m using spout or livelink to pass data back and forth between TD and UE on the same machine, it’s usually the first Engine setting adjustment I make.
However, it does nothing to guarantee sync, just eliminates a key, mechanism whereby the two applications run at drastically different rates. Experimenting with disabling ‘realtime mode’ in TD is probably also a useful avenue of exploration.
If you need actual sync, then your best bet would be to let UE actually drive your TD cook rate. I don’t have much experience with TouchEngine in UE but, I suspect that Tom was hoping to coax you towards doing all your TD-centric work there since this would allow TouchEngine’s clock to be controlled by UE.
As to Spout vs. NDI, I can see no good reason to use NDI locally on a system if Spout is a stable option. Spout uses a local shared GPU memory channel. It’s a shared buffer that is independently populated and sampled out of by the applications on either end. It provides no synchronization mechanism outside of memory safety stuff.
@bangnoise is your best bet for TouchEngine-UE expertise. I’m actually really interested in seeing where this goes.
edit: also, with regards to ‘throttling’ and frame rates, I’m pretty sure I was remembering it backwards. I think it was TD that would get throttled while trying to run in the background at 60fps when UE would have focus running at something like ~90.
Thanks for the answer! I will experiment with disabling realtime mode but I also think this is a more complex issue than it appears. The only reason I tested with NDI for a bit is because its a specific dedicated task (hardware decoding and reception) so i thought maybe it’d be less jankied up. But not at all lol. I’m also very eager to find a good solution as something smooth and stable would truly be a deal breaker!
Hello, I am also interested in seeing where this topic goes. I wanted to add couple of thoughts here.
@flowb already mentioned this, but I would like to emphasize that Spout has no synchronization mechanism. Therefore it could be causing you trouble with frame drops when used for such round-tripping.
The simplest option would probably be to isolate your TOP processing into a separate tox, and run it within TouchEngine-UE (like @bangnoise suggested). That way TouchEngine-UE provides you with necessary synchronization options (which are great, hats off). If you need to control what’s happening within TouchEngine from the main TD process, the best option would probably be some remote control over network protocols.
The problem with this workflow is that it is less interactive - as user can’t directly work on tox loaded within UE. Changes need to be saved into tox which then needs to be re-loaded within TouchEngine. I recall Derivative was exploring ways how to interactively work with TouchEngine processes, but I haven’t heard about the news in this area. Nevertheless it would be super cool to have this feature in the future.
In case you would like to go with the route you mentioned, this thread could be helpful.
I wanted to implement TextureShare plugin for TD, but so far I haven’t managed to dive into it. It would be cool to have as it provides texture sharing with synchronization mechanism. In case you have time and want to share texture directly between TD and UE with minimal latency and without frame drops, I would suggest using it. Hope this helps
thanks to you and @monty_python ! this is quite problematic for our workflow, as touch manages a whole lot of things with tops that would require a massive revamp. And since touch is supposed to be the last step before throwing out our content. I will dig into the texture sharing process, but I do not understand how I’d need to implement a custom plugin ish? Isnt DirectX In&Out the same as UE TextureSharing through directX?
oh okay you meant c++ top would permit me to implement syncing. Regarding the solution mentioned earlier, does it imply no solution for outputting an UE render target to td to output?
You can pass the Render Target directly to the TouchEngine process. There is a sample for that in the Sample project.
In the TouchEngine process / loaded .tox, you could use Spout to pass to your main TouchDesigner process. But again, you will lose any sync features at this point.
Thanks everyone for the answers - a lot of useful stuff above.
A custom UE TextureShare plugin would let you do UE → TouchDesigner directly, but it’s still not clear if it’s necessary to involve TouchDesigner at all (and it’d be a real pain if you don’t already have a C++ programmer with suitable experience to hand).
If possible, as we’ve each suggested, move all your TouchDesigner processing into TouchEngine-UE. If your final graphics output is to a window, you would have to send that back to UE for final display.
If you simply have to use TouchDesigner, using TouchEngine-UE only as a wrapper for Spout or NDI is overkill, and is needlessly introducing a third renderer. Hopefully there are UE Spout/NDI plugins which would do either of those directly without having to involve TouchEngine.
Whatever the solution, I’d note the documentation on Sync modes for TouchEngine-UE - it has to be configured to do single frame sync.
thanks for the answer. I figured TouchEngine-UE is kind of not suitable for our deployment, as it makes it way less versatile in the final stage to have the tox loaded into the packaged ue project for micro adjustments. I will try digging into an implementation of TextureShare through the c++ top as suggested. In case this doesn’t do the job, we will try going back to OWL plugin, but yet, this still has heavy sync problems especially from TD to UE where our emissive map being passed is struggling to catch every frames. Hopefully we come up to a solution. the TextureShare implementation would be quite a breakthrough for a longterm stable solution between the two softwares instances
Something that’s on my list to play with when I get back to Boston from Holidays is to test a workflow whereby you might have an externalized .tox/COMP within an engine instance and a mechanism for reloading.
This would let you abstract a network within a running TD instance, make modifications to it in another instance of TouchDesigner, then save over the .tox and trigger a reload in the instance running in TouchEngine.
It would certainly not be as flexible as being able to operate directly on the live network, but it’s a half measure perhaps.
The saveByteArray() and loadByteArray() methods on the COMP class would even potentially be a way to roll your own remote authoring and updating mechanisms.
yes, this would be handy indeep, but in case of a live event/performance, there is almost no room for such a manoeuver, could be a great compromise for other projects tho just to facilitate the workflow.
