SOLVED: WebRTC DAT `createOffer()` Not Initiating Connection & `onOffer` Callback Not Firing (Build 2023.12370)

capo_greco · May 28, 2025, 3:10pm

Hey team I have been working with Gemini and Claude to try to get WebRTC connectivity working, to no avail. I had Gemini prepare a report for this forum:

Hi everyone,

I’m working on setting up a WebRTC connection between TouchDesigner (64-Bit Build 2023.12370 on macOS) and a web browser, using an external Deno-based signaling server via WebSockets. I’m controlling the WebRTC DAT primarily through Python scripting.

Goal:
To have TouchDesigner initiate a WebRTC offer to a web client after the web client connects to the signaling server and TouchDesigner is notified via WebSocket.

Current Setup Overview:

Signaling Server: External Deno server (WebSocket based).
TouchDesigner Network (/project1/webrtc_complete/):
- websocket1 (WebSocket DAT): Connects to the Deno server. Its “Callbacks DAT” points to simple_handler_dat.
- webrtc1 (WebRTC DAT): Intended to handle the WebRTC peer connection. Its “Callbacks DAT” parameter points to simple_handler_dat.
- simple_handler_dat (Text DAT): Syncs to an external Python file (simple_handler_core.py). This script contains all WebSocket and WebRTC callbacks.
- Video/Audio Stream Out TOPs/CHOPs (intended to be connected later).

Problem:
When webrtc1.createOffer('connectionId') is called from the Python script (triggered by a WebSocket message indicating a new peer), the following happens:

The createOffer() Python method call itself executes without raising any Python exceptions.
However, no new row appears in the output table of the webrtc1 DAT. This table is expected to show active connections and their states.
Consequently, the onOffer(webrtcDAT_op, connectionId, localSdp) callback function (defined in simple_handler_dat / simple_handler_core.py) is not being triggered.

This prevents the offer SDP from being generated/retrieved and sent to the web client, thus blocking the WebRTC connection.

Key Diagnostic Steps & Findings:

WebRTC DAT Parameters (webrtc1 and a fresh test_webrtc):
- “Callbacks DAT” (callbacks): This parameter is confirmed to be correctly pointing to the Text DAT containing the Python callbacks. We’ve verified this by Python introspection (op('...').par.callbacks.eval()).
- “Active” (active): This parameter is ON.
- “Reset” (reset): Pulsing this button (or toggling Active off/on) does not resolve the issue.
- “Signaling State” Parameter: We have been unable to locate an explicit “Signaling State” parameter (with options like “External (Script)”) on the WebRTC DAT’s parameter dialog in this build (2023.12370). The available parameter pages are “Connection,” “ICE,” and “Common.” (A screenshot of the webrtc1 DAT’s “Connection” page parameters might be useful to include in the forum post).
Minimal Test Case:
- Created a new WebRTC DAT (test_webrtc) and a new Text DAT (test_callbacks).
- test_callbacks DAT contains only minimal stub functions, including:
```
# In test_callbacks DAT (Mode: Module, Language: Python)
print("DEBUG: test_callbacks DAT PARSED")
def onOffer(webrtcDAT_op, connectionId, localSdp):
    print(f"DEBUG: >>> MINIMAL onOffer FIRING for {connectionId}!!!")
    # webrtcDAT_op.setLocalDescription(connectionId, 'offer', localSdp) # Commented out for initial fire test
    return
```
- test_webrtc.par.callbacks is set to point to test_callbacks.
- test_callbacks Text DAT’s “Content Language” is Python. (It does not have a “Mode” parameter on its Common page in this build, only “Content Language”, “Edit/View Extension”, etc.).
- Executing op('/project1/webrtc_complete/test_webrtc').createOffer('someTestID') from the Textport runs without Python error.
- Result: Still no new row in test_webrtc’s output table, and the MINIMAL onOffer FIRING message does not appear in the Textport.
Python Script (simple_handler_core.py / simple_handler_dat):
- The script successfully initializes (top-level prints are seen).
- WebSocket DAT callbacks (onConnect, onReceiveText, etc.) in the script are working correctly.
- The script correctly identifies the path to webrtc1.
- When a peer-connected message is received via WebSocket, the script successfully calls webrtc1.createOffer(new_connection_id).
- The Python parameter access issues (e.g., .par.connected, .par.netaddress) in the script have been resolved.

Summary of Current Dead End:
The createOffer() method on the WebRTC DAT seems to be a “no-op” in this build/configuration: it doesn’t raise a Python error, but it also doesn’t appear to initiate any internal WebRTC offer process (as evidenced by no change in its output table) and subsequently doesn’t trigger the onOffer Python callback, even with a minimal setup and correctly assigned Callbacks DAT. The absence of an explicit “Signaling State: External (Script)” parameter is noted.

Questions for the Forum:

For TouchDesigner Build 2023.12370, if the WebRTC DAT does not have an explicit “Signaling State” parameter, how is “external script-based signaling” (using Python callbacks for onOffer, onIceCandidate, etc.) enabled? Is setting the “Callbacks DAT” parameter sufficient, or is there another mechanism/parameter?
Under what conditions would webrtcDAT.createOffer('id') execute without Python error but fail to add a new connection entry to the DAT’s output table and fail to trigger the onOffer callback?
Are there any known issues or specific configurations required for WebRTC DAT Python callbacks in this build version?
Could there be an issue if the me object or pathing to the WebSocket DAT within the onOffer callback (when defined in an external module loaded into a Text DAT) is problematic, even if the callback itself isn’t firing? (Though this seems less likely if the callback doesn’t even start).

Any insights or suggestions would be greatly appreciated. I can provide the simple_handler_core.py script and a minimal .toe file if helpful.

Thanks!

example.1.toe (7.9 KB)

# simple_handler_core.py
import json
import uuid

print(f"==============> !!!!! simple_handler_core.py TOP LEVEL EXECUTING.")

# Path setup for webrtc_dat_path (assuming 'webrtc1' is the name of the WebRTC DAT)
try:
    script_dat_operator = me 
    parent_comp = script_dat_operator.parent()

    stored_path = script_dat_operator.storage.get('webrtc_dat_path')
    if stored_path and op(stored_path):
        webrtc_dat_path = stored_path
        print(f"ℹ️ Using WebRTC DAT path from storage: {webrtc_dat_path}")
    else:
        sibling_webrtc_op = parent_comp.op('webrtc1') 
        if sibling_webrtc_op:
            webrtc_dat_path = sibling_webrtc_op.path
            print(f"ℹ️ Using WebRTC DAT path (sibling to {script_dat_operator.name}): {webrtc_dat_path}")
        else:
            webrtc_dat_path = '../webrtc1' 
            print(f"⚠️ Could not find 'webrtc1' as sibling to {script_dat_operator.name} or in storage. Falling back to relative path from {parent_comp.path}: {webrtc_dat_path}")
            if not parent_comp.op(webrtc_dat_path): 
                 print(f"❌ CRITICAL: Fallback path {webrtc_dat_path} for WebRTC DAT (relative to {parent_comp.path}) is also invalid.")
except Exception as e:
    print(f"❌ ERROR during WebRTC DAT path setup: {e}")
    webrtc_dat_path = "/PATH/TO/NONEXISTENT/WEBRTC_DAT" 

# Global state
webrtc_clients = {} 
local_client_id = "TouchDesigner_Client" 

# --- WebSocket Callbacks ---

def onConnect(dat): # 'dat' is the WebSocket DAT
    try:
        # CORRECTED: Use .par.netaddress (lowercase)
        print(f"✅ WebSocket '{dat.name}' connected to {dat.par.netaddress.eval()}")
    except Exception as e:
        print(f"Error in onConnect accessing par.netaddress: {e}") 
        print(f"✅ WebSocket '{dat.name}' connected (address parameter info unavailable).")


def onDisconnect(dat): 
    print(f"❌ WebSocket '{dat.name}' disconnected.")

def onReceivePing(dat, contents): 
    print(f"🏓 Ping received from WebSocket. Responding.")
    dat.sendPong(contents)

def onReceiveText(dat, rowIndex, message): 
    global webrtc_clients, local_client_id
    script_dat_operator = op(me.path) 

    print(f"DEBUG: WebSocket received raw message: '{message}'")
    
    try:
        msg_data = json.loads(message)
        msg_type = msg_data.get('type')
        sender_peer_id = msg_data.get('from')
        print(f"📨 Received WebSocket message of type '{msg_type}' from '{sender_peer_id}'")

        webrtc_op = op(webrtc_dat_path) 
        if not webrtc_op:
            print(f"   ❌ ERROR: WebRTC DAT at path '{webrtc_dat_path}' not found in onReceiveText!")
            return

        if msg_type == 'peer-connected':
            connected_peer_id = msg_data.get('data', {}).get('peerId')
            peer_type = msg_data.get('data', {}).get('peerType')
            print(f"DEBUG: Handling 'peer-connected' for server_peer_id: {connected_peer_id}, peerType: {peer_type}")
        
            if peer_type == "web":
                if connected_peer_id in webrtc_clients:
                    print(f"   ⚠️ Peer {connected_peer_id} already has an entry. Re-initializing offer.")
                
                print(f"   Paired with web peer: {connected_peer_id}")
                print(f"   Starting WebRTC offer process...")
                
                td_connection_id = str(uuid.uuid4()) 
                print(f"   Generated new TD WebRTC Connection ID: {td_connection_id}")
        
                webrtc_clients[connected_peer_id] = {
                    'td_connection_id': td_connection_id,
                    'state': 'initiating_offer',
                    'peer_type': peer_type
                }
                print(f"DEBUG: webrtc_clients updated: {webrtc_clients}")
        
                try: 
                    websocket_dat_op = op(dat.path) # 'dat' is the websocket DAT passed to onReceiveText
                    if websocket_dat_op:
                         # CORRECTED: Use .par.connected (lowercase)
                         print(f"DEBUG: WebSocket state BEFORE createOffer: {websocket_dat_op.par.connected.eval()}")
                    else:
                        print(f"DEBUG: Could not find WebSocket DAT '{dat.path}' for pre-offer state check.")

                    webrtc_op.createOffer(td_connection_id) 
                    print(f"   ✓ createOffer called for TD Connection ID {td_connection_id}")
                    
                    if websocket_dat_op:
                         # CORRECTED: Use .par.connected (lowercase)
                        print(f"DEBUG: WebSocket state AFTER createOffer: {websocket_dat_op.par.connected.eval()}")

                except Exception as e:
                    print(f"❌ ERROR calling createOffer for TD Connection ID {td_connection_id}: {e}")
                    import traceback 
                    traceback.print_exc()
            else:
                print(f"   Ignoring 'peer-connected' from non-web peer: {connected_peer_id}, type: {peer_type}")
        
        elif msg_type == 'answer':
            if not sender_peer_id:
                print(f"   ❌ ERROR: Received 'answer' without 'from' field. Cannot route. Message: {message}")
                return

            if sender_peer_id in webrtc_clients:
                td_conn_id = webrtc_clients[sender_peer_id].get('td_connection_id')
                sdp_answer = msg_data.get('sdp')
                if td_conn_id and sdp_answer:
                    print(f"   Received answer from '{sender_peer_id}' for TD Connection ID {td_conn_id}")
                    webrtc_op.setRemoteDescription(td_conn_id, 'answer', sdp_answer) 
                    print(f"   ✓ Remote description (answer) set for TD Connection ID {td_conn_id}")
                    webrtc_clients[sender_peer_id]['state'] = 'answer_received'
                else:
                    print(f"   ❌ ERROR: Missing td_connection_id or SDP in 'answer' from {sender_peer_id}")
            else:
                print(f"   ❌ WARNING: Received 'answer' from unknown or unmapped peer: {sender_peer_id}")

        elif msg_type == 'ice-candidate':
            if not sender_peer_id:
                print(f"   ❌ ERROR: Received 'ice-candidate' without 'from' field. Message: {message}")
                return

            if sender_peer_id in webrtc_clients:
                td_conn_id = webrtc_clients[sender_peer_id].get('td_connection_id')
                candidate_payload = msg_data.get('data') 
                
                if td_conn_id and candidate_payload and 'candidate' in candidate_payload:
                    print(f"   Adding ICE candidate from '{sender_peer_id}' for TD Connection ID {td_conn_id}")
                    webrtc_op.addIceCandidate(
                        td_conn_id, 
                        candidate_payload.get('candidate'),
                        candidate_payload.get('sdpMLineIndex'), 
                        candidate_payload.get('sdpMid')
                    )
                    print(f"   ✓ ICE candidate added for TD Connection ID {td_conn_id}")
                else:
                    print(f"   ❌ ERROR: Missing td_connection_id or valid candidate payload in 'ice-candidate' from {sender_peer_id}. Payload: {candidate_payload}")
            else:
                print(f"   ❌ WARNING: Received 'ice-candidate' from unknown or unmapped peer: {sender_peer_id}")
        
        elif msg_type == 'peer-disconnected':
            disconnected_peer_id = msg_data.get('data', {}).get('peerId')
            print(f"   Peer '{disconnected_peer_id}' reported disconnected by server.")
            if disconnected_peer_id in webrtc_clients:
                td_conn_id = webrtc_clients[disconnected_peer_id].get('td_connection_id')
                if td_conn_id:
                    print(f"   Closing WebRTC connection for TD Connection ID {td_conn_id}")
                    webrtc_op.closeConnection(td_conn_id)
                del webrtc_clients[disconnected_peer_id]
                print(f"   Removed '{disconnected_peer_id}' from webrtc_clients. Current clients: {list(webrtc_clients.keys())}")
            else:
                print(f"   Peer '{disconnected_peer_id}' was not in our active list.")

    except json.JSONDecodeError:
        print(f"DEBUG: Received non-JSON WebSocket message: '{message}'")
    except Exception as e:
        print(f"❌ Unhandled Error in onReceiveText: {e}")
        import traceback
        traceback.print_exc()

# --- WebRTC Callbacks ---

def onOffer(webrtcDAT_op, connectionId, localSdp): 
    global webrtc_clients, local_client_id
    script_dat_operator = op(me.path) 

    print(f"DEBUG: >>> onOffer FIRING for TD Connection ID: {connectionId}")
    print(f"DEBUG:       Offer SDP (first 60 chars): {localSdp[:60]}...")

    try:
        print(f"DEBUG:       Attempting to setLocalDescription within onOffer for {connectionId}...")
        webrtcDAT_op.setLocalDescription(connectionId, 'offer', localSdp)
        print(f"DEBUG:       ✅ setLocalDescription called within onOffer for {connectionId}.")
    except Exception as e:
        print(f"❌ ERROR calling setLocalDescription in onOffer for {connectionId}: {e}")
        import traceback; traceback.print_exc(); return 

    target_server_peer_id = None
    for server_pid, conn_data in webrtc_clients.items():
        if conn_data.get('td_connection_id') == connectionId:
            target_server_peer_id = server_pid; break
    
    if not target_server_peer_id:
        print(f"❌ ERROR: In onOffer, could not find target_server_peer_id for TD Connection ID '{connectionId}'. Current webrtc_clients: {webrtc_clients}")
        return
    
    websocket_op = script_dat_operator.parent().op('websocket1') 
    if not websocket_op:
        print(f"❌ ERROR: WebSocket DAT 'websocket1' (sibling to {script_dat_operator.name}) not found in onOffer.")
        return

    message_to_send = {"type": "offer", "sdp": localSdp, "to": target_server_peer_id, "from": local_client_id }
    try:
        print(f"DEBUG:       Attempting to send 'offer' to server_peer_id '{target_server_peer_id}' via WebSocket '{websocket_op.path}'...")
        websocket_op.sendText(json.dumps(message_to_send))
        print(f"DEBUG:       ✅ 'offer' sent to '{target_server_peer_id}'.")
        if target_server_peer_id in webrtc_clients:
             webrtc_clients[target_server_peer_id]['state'] = 'offer_sent'
    except Exception as e:
        print(f"❌ ERROR sending 'offer' in onOffer: {e}")
        import traceback; traceback.print_exc()

def onAnswer(webrtcDAT_op, connectionId, localSdp):
    global webrtc_clients, local_client_id
    script_dat_operator = op(me.path)
    print(f"DEBUG: >>> onAnswer FIRING for TD Connection ID: {connectionId}")
    try:
        webrtcDAT_op.setLocalDescription(connectionId, 'answer', localSdp)
    except Exception as e: print(f"❌ ERROR setLocalDescription in onAnswer: {e}")

    target_server_peer_id = None 
    for server_pid, conn_data in webrtc_clients.items():
        if conn_data.get('td_connection_id') == connectionId:
            target_server_peer_id = server_pid; break
    if not target_server_peer_id: return

    websocket_op = script_dat_operator.parent().op('websocket1')
    if not websocket_op: return
        
    message_to_send = {"type": "answer", "sdp": localSdp, "to": target_server_peer_id, "from": local_client_id}
    try:
        websocket_op.sendText(json.dumps(message_to_send))
        print(f"DEBUG:       ✅ 'answer' sent to '{target_server_peer_id}'.")
        if target_server_peer_id in webrtc_clients: webrtc_clients[target_server_peer_id]['state'] = 'answer_sent'
    except Exception as e: print(f"❌ ERROR sending 'answer': {e}")

def onIceCandidate(webrtcDAT_op, connectionId, candidate, sdpMLineIndex, sdpMid):
    global webrtc_clients, local_client_id
    script_dat_operator = op(me.path)

    print(f"DEBUG: >>> onIceCandidate FIRING for TD Connection ID: {connectionId}")
    print(f"DEBUG:       Candidate: {candidate[:60]}..., sdpMLineIndex: {sdpMLineIndex}, sdpMid: {sdpMid}")

    target_server_peer_id = None
    for server_pid, conn_data in webrtc_clients.items():
        if conn_data.get('td_connection_id') == connectionId:
            target_server_peer_id = server_pid; break
    if not target_server_peer_id:
        print(f"❌ ERROR: In onIceCandidate, could not find target_server_peer_id for TD Connection ID '{connectionId}'. webrtc_clients: {webrtc_clients}")
        return

    websocket_op = script_dat_operator.parent().op('websocket1')
    if not websocket_op:
        print(f"❌ ERROR: WebSocket DAT 'websocket1' (sibling to {script_dat_operator.name}) not found in onIceCandidate.")
        return

    message_to_send = {
        "type": "ice-candidate",
        "data": { "candidate": candidate, "sdpMLineIndex": sdpMLineIndex, "sdpMid": sdpMid },
        "to": target_server_peer_id, "from": local_client_id
    }
    try:
        print(f"DEBUG:       Attempting to send 'ice-candidate' to server_peer_id '{target_server_peer_id}'...")
        websocket_op.sendText(json.dumps(message_to_send))
        print(f"DEBUG:       ✅ 'ice-candidate' sent to '{target_server_peer_id}'.")
    except Exception as e:
        print(f"❌ ERROR sending 'ice-candidate' in onIceCandidate: {e}")
        import traceback; traceback.print_exc()

def onConnectionStateChange(webrtcDAT_op, connectionId, newState):
    print(f"🔗 WebRTC Connection State Change for TD Connection ID {connectionId[:8]}... New State: '{newState}'")
    server_peer_id_of_connection = None
    for p_id, data in webrtc_clients.items():
        if data.get('td_connection_id') == connectionId:
            data['state'] = newState; server_peer_id_of_connection = p_id; break
    if newState == 'connected': print(f"✅ WebRTC Connection ESTABLISHED for TD Connection ID {connectionId[:8]} (Server Peer ID: {server_peer_id_of_connection})")
    elif newState == 'failed': print(f"❌ WebRTC Connection FAILED for TD Connection ID {connectionId[:8]} (Server Peer ID: {server_peer_id_of_connection})")
    elif newState == 'closed' or newState == 'disconnected':
        print(f"ℹ️ WebRTC Connection CLOSED/DISCONNECTED for TD Connection ID {connectionId[:8]} (Server Peer ID: {server_peer_id_of_connection})")

def onIceConnectionStateChange(webrtcDAT_op, connectionId, newState):
    print(f"🧊 ICE Connection State Change for TD Connection ID {connectionId[:8]}...: '{newState}'")

def onIceCandidateError(webrtcDAT_op, connectionId, errorText):
    print(f"❌ ICE Candidate Error for TD Connection ID {connectionId[:8]}...: {errorText}")

def onNegotiationNeeded(webrtcDAT_op, connectionId):
    print(f"⚠️ WebRTC Negotiation Needed for TD Connection ID {connectionId[:8]}...")

def onTrack(webrtcDAT_op, connectionId, trackId, trackType): 
    print(f"🎥 Track '{trackType}' (ID: {trackId}) received on TD Connection ID {connectionId[:8]}...")

def onRemoveTrack(webrtcDAT_op, connectionId, trackId, trackType):
    print(f"🚫 Track '{trackType}' (ID: {trackId}) removed on TD Connection ID {connectionId[:8]}...")

def onDataChannel(webrtcDAT_op, connectionId, channelName):
    print(f"📡 Data Channel '{channelName}' created on TD Connection ID {connectionId[:8]}...")

def onDataChannelOpen(webrtcDAT_op, connectionId, channelName):
    print(f"✅ Data Channel '{channelName}' OPENED on TD Connection ID {connectionId[:8]}...")

def onDataChannelClose(webrtcDAT_op, connectionId, channelName):
    print(f"❌ Data Channel '{channelName}' CLOSED on TD Connection ID {connectionId[:8]}...")

def onData(webrtcDAT_op, connectionId, channelName, data): 
    try:
        decoded_data = data.decode('utf-8')
        print(f"📥 Data received on channel '{channelName}' from TD Connection ID {connectionId[:8]}...: '{decoded_data}'")
    except UnicodeDecodeError:
        print(f"📥 Binary data received on channel '{channelName}' from TD Connection ID {connectionId[:8]}...: {len(data)} bytes")

# def check_connection_status_table():
#     # Optional: implement if needed for debugging the DAT table directly
#     pass

eric.b · May 28, 2025, 3:26pm

I think the missing piece here is webrtcDAT.openConnection(), which generates and initializes the local end of the peer connection and returns the uuid for use in subsequent methods such as webrtcDAT.createOffer(...). Ultimately, it is openConnection that adds the row to the table.

I think it might be helpful here if methods like createOffer return a warning if used with a uuid that hasn’t been initialized with openConnection.

capo_greco · May 29, 2025, 4:50am

Incredible - thank you @eric.b , that helped immensely.

I have a follow up problem, here