WebRTC DAT Tutorial

Hi all,

So I’m working on a project & assisting someone else in using TouchDesigner on some data visualizations. We’re hoping to utilize the new WebRTC features in the experimental build, but we realize this is very, very new and there’s limited documentation on it.

Would anyone be willing to drop a simple example that utilizes either the WebRTC DAT sending to a webpage (doesn’t need to be functional, just demonstrating how you go about setting it up) or using the palette components to stream to a webpage instead of from TD instance to TD instance. I’m unfortunately not so versed in WebRTC as a protocol, but would really love it if anyone more experienced might have a simple setup and deployment.

Thank you in advance!

Hey @luxnaut

They are no “simple example”.

WebRTC is not a plug n play technology, mostly because of the variety of scenarios, fragmented web tech stack, and various signaling APIs.

If your ultimate goal is to send from TD to a web page, you can use the Signaling Server that we ship in the Palette, the Signaling Client and the WebRTC components without having to write a line of code on TD’s end and you don’t have to study the WebRTC DAT.

The palette components are quite complete and should be flexible enough to extend easily with additional features.

That being said, if you start to dig into the code, I’d consider it advanced programming.

On the receiving end, the web page, this would all be custom code… but all the examples from MDN or Google are easy to adapt for your web app to exchange messages with TD’s signaling server. A good first step would be to have a webpage going where you have a signaling client (websocket) and signaling events being sent / received, managing a single MediaStream and a single RTCPeerConnection in the webapp.

That would be the most simple setup that I can think of at the moment.

Note that you cannot just copy paste the Javascript from MDN. You need to adapt it slightly for it to understand TD’s Signaling API.

The WebRTC documentation in the wiki is quite extensive with details about specific use cases that we do not cover in the palette components.

I hope this can get you started.

Best,
Michel

Hi @JetXS,

I am working with @ahfowler (who’s more familiar with web dev, and I with TD) in creating a simple connection between a website application and TD. However, we’re a little unsure what will be exposed for @ahfowler to use from TD’s signaling server.

We are trying to stream a custom-visualization video generated with real-time data in TD to a website, and the website will not be sending anything back.

We’re trying to find the appropriate example in Google’s [WebRTC library] (WebRTC samples), but are not sure if TD would be providing a video/canvas element. In other words, in this documentation, what would be the TD equivalent of HTMLVideoElement?

Any .toe file that you might be able to provide just to show functionality of the palette component would also be super appreciated!

Thank you in advance!

1 Like

Hey @luxnaut

TD doesn’t provide a video / canvas element. TD is pushing a / many tracks over an RTCPeerConnection.

An RTCPeerConnection gets in a connected state when the connection is established following the signaling process: when it went through the exchange of offers / answers / candidates.

In JS, you would first setup your own signaling client to exchange signaling messages with TouchDesigner. Using WebSocket in our case if you stick to TD’s own signaling server.

All the messages are documented in the Wiki and JSON Schema is available to test your messages.

Once your signaling client is running, you can establish an RTCPeerConnection.

As soon as the state of your RTCPeerConnection is in a “connected” state, a track event should fire. From the track event, you can get the WebRTC Track to add to a MediaStream, and push that MediaStream to the srcObject of a video element.

Having the palette tools running in TD is fairly easy.

You don’t need multiple computers or even multiple instances running.

Drag n drop the signalingServer from the palette - > Change its Active par to ON.

Drag n drop 2 signalingClient-s from the palette → Change their Active par to ON. Change their Forward to Subscribers par to ON.

Drag n drop 2 WebRTC COMP-s from the palette → Assign the first signalingClient to the first WebRTC COMP, repeat with second.

Now if you look at the signalingServer, you’ll see 2 signalingClients showing up.

On each signalingClient, you will see the other signalingClient.

On each WebRTC COMP, you will see the other client (the client that represents a signalingClient which is not the one assigned to the WebRTC COMP you are looking at)

You just have to go in active mode to start a call.

With all that in mind, what you have to achieve in your web page / project is for a client to show up in the signalingServer, for the WebRTC COMP to be able to call it (start an RTCPeerConnection and make an offer).

I hope this helps and get you going,

Best,
Michel

Hi @JetXS,

Thank you so much for your help thus far.

Currently, we have a signaling channel from AWS Kinesis Video Streams and we want to connect TouchDesigner as a peer to this signaling channel. AWS has SDKs for Web/Android/iOS applications (https://docs.aws.amazon.com/kinesisvideostreams-webrtc-dg/latest/devguide/webrtc-sdks.html) to connect to the signaling channel, but I’m unsure what SDK to use for TouchDesigner since it doesn’t seem to fit into any of the options. Do you have any suggestions?

If not, that is okay-- I decided to look into connecting the website to TouchDesigner’s signaling server. However, we were running into issues because if I serve my website on port 1234, then it wouldn’t be able to connect to the signaling server also on port 1234.

Is there a way to connect a signaling client, such as my website and its WebSockets, to a signaling server on a different port?

Thank you in advance!
Azaria

Hey @ahfowler,

I am not familiar enough with AWS Kinesis, but it seems that the pages you should be looking at are the following, rather than the SDK page.

You want to solve the signaling process from TouchDesigner, to the signaling server provided by AWS. You have to write your own Signaling Client, in TouchDesigner, to join the AWS signaling session. What the doc seem to say is that it can be done in WebSocket. This would be implemented in Python in TouchDesigner and rely on the https://docs.derivative.ca/WebSocket_DAT

The Signaling Server provided in TouchDesigner and the Signaling Client are on port 9980 by default, not 1234. You can change this port.

image

Best,
Michel

Hey, WebRTC is starting to mature. Here’s a fairly standard signalling type coming out:

Can we get this in TouchDesigner please?
Also hardware video encoding please!

Bruce

I followed these steps and got the components talking to each other on one machine…

Is it equally as easy to get a touchdesigner client on another machine across the internet talking? Is it just a matter of putting the right thing in the Host parameter in the remote signalingClient?

Hey @ianl

It should be yes. If you are having issues it would likely be caused by a complex network configuration or firewalls.

Best,
Michel

Hi,

I tried your tutorial within one process or between two processes and there was a delay of about 3 seconds in sending and receiving.

I’m hoping the delay is within a few hundred milliseconds, what could be the problem?

I tried it with 2022.26590 on Windows 11, Ryzen 5900 CPU, 16GB RAM, RTX 3060 GPU.

Thanks,

Akihiro

Hey @comoc

While I am hoping we can improve the latency in future updates, I cannot get to 3 seconds which seems to be way too high.

Are you observing 3 seconds over the network or between 2 local processes?

Are you dropping frames in the sending or receiving process ?

Thanks,
Michel

Hi @JetXS ,

Here is the video how I measured the delay within a single process.

Please note especially the hand movements in the left and right webRTC COMPs.

Even with two local processes, the situation was the same.

Thanks,

1 Like

Hey @comoc

Note that you are dropping frames.

We are looking into improving the delay nonetheless but it might be that your test case is not helped by those frames being dropped.

Will keep you updated when we have an update to share.

Best,
Michel

Hi @JetXS I’m learning webRTC in TouchDesigner, here is my first connection between different touchdesigner application, but it can’t recevie new video source in other app. so do I miss thing?

Hey @Wingto

How did you reference the new moviefilein1? Did you use the track manager ?

Best,
Michel

this I drop the new Top into the track manager

In your receiver, can you click the dropdown of the stream you receive and see how many tracks are here ? Could be a cooking bug and the menu needs to be updated just to select the new trackId.

I just gave it all a try and was able to get the new track to display with no issues.

Hey @bwheaton

Thanks for sharing this. Just a quick message to acknowledge that I went over those docs, and some extras.

We will discuss this internally.

Right now the priority is hardware encoding + some extra tools for the community around the “regular” signaling workflow.

Best,
Michel

1 Like

I met a other trouble, take a look what I made misake please :>

You might have gone a bit too fast and not let enough time for the WebRTC COMP to subscribe to the signalingClient it is working :handshake: with.

The second time you press to trigger a new call but you already have an ongoing “tentative call” ongoing. What you can do is hit “Reset” on the WebRTC COMP, or in the bottom part of the WebRTC COMP viewer, close the current “failing” call.

Then call again, and it should hopefully work.

This workflow will be improved and more reliable in a future update.