MediaPipe Gesture Control for Blender FBX Lights in TouchDesigner

General TouchDesigner Discussion
1

Hey everyone,
I’m working on an interactive TouchDesigner setup and wanted to ask about the best workflow for controlling Blender FBX lights using MediaPipe hand gestures.

Current pipeline:
Blender scene (with lights) → FBX → TouchDesigner → MediaPipe gesture tracking → realtime light control

Inside TD, the FBX imports correctly with geometry/cameras/lights.

My goal is to use gestures like:

  • open hand = increase intensity

  • fist = lights off

  • pinch = dim/brighten

  • hand position = move light

A few questions:

  • Is it better to control the imported FBX lights directly, or recreate native TD Light COMPs?

  • What’s the cleanest workflow for connecting MediaPipe data to light parameters?

  • Any recommendations for stable MediaPipe setups/components in TD?

    I attached this reference for the type of interaction/look I’m aiming for:
    https://www.instagram.com/p/DXCvTsEkrVe/

If anyone has a tutorial, GitHub project, example file, or breakdown for this kind of interactive setup, please share. Thanks!

2

Hello,

I found very dificult to animate properly an imported FBX in TD. In my case the movement as analysed by Kinect skeleton. Invert kinematics, proper rotation/displacement etc. are very complex to obtain. If the mediapipe info are only for lightning, its doable.

I tried the mediapipe TD plugin

But I found it very difficult to set and to use. I had much better result with face track Chop and body track Chop but you need a nVidia card > 10xx

I found easier to animate and light inside Blender and send the resulting render via spout. Unfortunately its very difficult to send image with the alpha layer from Blender trough spout and I found a solution using a very green background in Blender and chromaKeying in TD.

The quality of 3D render by Blender/eevee is a lot better than in TD and there is a very good OSC plugin in Blender allowing you to send mediapipe infos to Blender.

3

Thank you, this is really helpful.
I’m actually more focused on controlling lighting and atmosphere rather than full character animation/IK, so maybe the TD workflow could still work for my case.

Your Blender + Spout + TD pipeline sounds very interesting though. Could you share a bit more about how you connected everything together? Especially:

  • how you sent MediaPipe/OSC data into Blender

  • and your Spout workflow between Blender and TD

Would love to understand your setup more because the visual quality you mentioned from Eevee is also important for the direction I’m aiming for.

4
1 Like