Stunned by webRTCPanel

I have been playing with new webRTCPanel & webRTCPanelRcv and I am amazed by this concept.

Please may I ask if there are some future plans for building a website that would be designed to receive a WebRTC stream, interact with TD’s panel and would support TD’s Signaling API?

If there aren’t such plans, please what would you recommend as best approach for building such website? I guess it would be fairly complex thing to do, am I right? Thanks :slight_smile:

4 Likes

Hey @monty_python,

thanks for the feedback!

A web demo is being worked on as we speak yes.

Nothing really that I’d call production ready (it’s all quite custom) but enough to showcase the concept and have a stable connection from TD to the browser, with interactivity.

This will be a JS + React single page.

This is not something complex to do from scratch if you have a JS dev on your team.

Best,
Michel

6 Likes

That sounds really amazing. I will be looking forward to playing with it. The amount of possibilities is truly raising exponentially with these kind of tools :slight_smile:

Please may I ask about codecs and their encoding / decoding?

I am curious what codec is currently being used for webRTC video stream between two TDs? Based on my quick internet search I have noticed people usually take a look at SDP messages to figure this out. Please is there some convenient way of taking a look at these messages in TD? I guess I can catch them using Wireshark, but I wanted to ask in case there is some better way of doing so.

Because the Video Stream In / Out TOPs and Audio Stream In / Out CHOPs (some of the underlying operators of this component) are relying on Libwebrtc’s own encoder / decoder and the library does not support hardware encoding / decoding

I have noticed this in docs (mentioned under section Known issues in experimental). May I ask if this is also the case for current official builds? I am curious whether there might be a possibility to use hardware acceleration for encoding / decoding H264 / VP9 in the future?
Thanks

Here are details about WebRTC Codecs: Codecs used by WebRTC - Web media technologies | MDN

The codec currently used for video is VP8, with a fallback if needed to H264 if I am correct.

Looking at the SDP should give you clues indeed.

HW encoding and decoding are in the works, but no ETA.

Best,
Michel

1 Like

Hey @monty_python and others,

A web demo is now available at the following repository GitHub - TouchDesigner/WebRTC-Remote-Panel-Web-Demo: A simple React project that can receive a WebRTC video stream from TouchDesigner and send back mouse and keyboard events.

The new TouchDesigner release comes with a TLS tweak that will enable all this to work over a secured WebSocket and the online demo available here: TD WebRTC Web Demo 🍌

And we also released an introductory article:

https://derivative.ca/community-post/experience-webrtc-and-webrtc-remote-panel-new-webrtc-remote-panel-web-demo/67004

Best,
Michel

6 Likes

Wow @JetXS that is pretty sweet, great work!

1 Like

@JetXS this is amazing! I will dive into testing it as soon as possible :slight_smile:

1 Like

@JetXS I have just played with this new Web Demo and it looks great. :+1:

I have tried this with Microsoft Surface + Chrome and I really how CPU & GPU usage is very low. This is great for all sorts of low power devices. Especially in scenarios where panel isn’t cooking for some time, the usage is absolutely minimal (as no video is being sent over the network - thus no video decoding is needed). This is brilliant. :slight_smile:

I wanted to ask if it might be possible to add option for making the video fullscreen in the future? I have tried using controls provided by Chrome to maximize it, but there seems to be a problem with video being flipped once maximized. Also once it is maximized, I can’t seem to permanently disable these controls at the bottom. I thought maybe there would be a way in JavaScript to get this video into some clean fullscreen mode, but I have never worked in that area so I am just guessing here.

Also I have noticed that while my mouse on Surface worked, I wasn’t able to “draw” using finger or pen. It seemed like Chrome was consuming these events rather than WebRTC. I guess this is just the way Chrome works by default (again I am not sure if there is some control over this or not, but I thought I might mention it here).

One final thing I have noticed during my experiments was that clicking a button doesn’t work exactly like it should. This isn’t related to Web Demo, but rather to WebRTCRemotePanel COMP. For example when I have a button set to momentary type, after clicking on it, it remains “clicked” until I move mouse away from it.

Also I am not sure if this might help somehow, but TD have crashed on me once during these experiments. I am attaching project with dump in case you might find something there. I have been running NodeJS locally, but I have installed newer version than was recommended on github (18.13.0 LTS), which might have been a bad idea. :slight_smile:
TouchDesignerCrash.2022.31030.1.dmp (319.2 KB)
CrashAutoSave.NewProject.1.toe (139.1 KB)

Hey @monty_python

Thanks for the feedback here.

Glad you like the performances at the moment. I found that the CPU (lack of hardware encoding / decoding) was causing the WebRTC pipeline to not be so scalable (when ideally, it should). I hope we can improve on this in the future. For the time being a 1080p remote panel should be handled nicely.

The issue is that if you go fullscreen with Chrome, it doesn’t apply the CSS styling anymore that is on the video container and flipping the receiving stream. I’ll add this as an RFE but no ETA. It likely involves some frontend JavaScript as well, indeed. It might be worth it tweaking this on your end or having a web developer that you might work with look into this.

Yes, the touch events are not all currently supported, the logic is a bit different than with a mouse and I believe they even have their own handlers in the browser API, which is not implemented. I’ll add this as an RFE but again no ETA and this might be something you want to look into on your end with a web developer.

The last case sounds more like a bug, I will give it another try and look into it.

Regarding the crash: TD Shouldn’t crash just because you have a different NodeJS version installed. Maybe for some reason the negotiation would fail, but other than that, it shouldn’t trigger a crash so this needs to be looked into.

Do you know how to reproduce it ? Or was it just plain random ?

Thanks again for the feedback,
Best,
Michel

Yeah, hardware encoding on TD side would be great, however I believe Chrome on my Surface was doing hardware decoding - which resulted in low performance overhead on on this side of things.

That’s right, I am not sure when I will get to it, but I was thinking it would be best to start looking into JavaScript to be able to do this kind of stuff. :slight_smile:

Unfortunately I don’t recall what exactly I did, but I think it was related to hitting start / end buttons on Chrome side. I feel like I have clicked on some of these options and TD crashed right away. I definitely didn’t do anything with TD at that point in time. I know this doesn’t help you much - I will try to remember this better in case it happens again.

The crash points to a panel being removed in TD.

Just to be clear, where was TD running ? Not on the Surface Pro I’m guessing ?

I’ll see if I can reproduce.

Interesting, I am quite sure I haven’t touched TD at that moment.

TD was running on my notebook where I do all TD-related work. Surface Pro was just running Chrome with Web Demo site opened (which was also served by NodeJS running on my notebook).

Before TD crashed, I have clicked on one of the buttons in Chrome (not sure whether it was Start or End, but I feel like it was one of them).

Hey @monty_python

I’ve tracked this down to an event not being passed in a react component.

If you run the code locally and have the source already, it’s an easy fix:

src/components/MediaPanel.js:Line 71 add onMouseUp={ sendMouseData }

And that should fix it. Tested on my end and it looked good so far.

Please let me know if that helps and I’ll push a fix to the online demo.

Best,
Michel

Edit: Fix being rather straightforward, this was already pushed to the live demo.

@JetXS Thank you very much for such a fast fix :slight_smile:
In case I will notice any issues I will let you know, but that crash yesterday happened only after couple hours of experimenting, so I guess it won’t be so easy to test. But if I won’t see any crash during longer testing I will consider this fixed. :+1:

1 Like