Object placement tracking using Kinect and MediaPipe

Beginners
,
1

Hi everyone, I’ll get straight to the point.

I’m working on a project where I use data from a Kinect sensor to visualize people as particle systems, essentially recreating what the Kinect sees using particles. I’m now trying to combine this with the MediaPipe plugin.

The idea is to use MediaPipe’s pose tracking—specifically the X and Y coordinates of the nose—to attach a 3D object (like a marker or accessory) that stays fixed to the person’s nose and moves with them. The goal is to combine the particle output from the Kinect with the nose-tracked object from MediaPipe in a Composite TOP, so both visualizations overlay correctly in real time.

However, I’m running into an issue: when I use the nose’s X and Y coordinates from MediaPipe to position the 3D object, the object moves faster than the person and ends up misaligned.

Does anyone have any tips or suggestions on how to fix this issue? Any help would be greatly appreciated. Thanks!
Testing_pose_tracking.toe (591.1 KB)

2

I haven’t used MediaPipe in quite some time but it’s most likely the values it outputs exceed the range that is needed in TouchDesigner. I see in your example you’re extracting the raw nose data - best bet would be to chuck in a Math CHOP in between and use the Range tab to remap an incoming range to a new range - eg. you might be getting a value from 0 - 1 but you need it to be from -0.5 - 0.5.

As a side note (if you have compatible hardware), pose tracking can be done by the Kinect as well and will be more efficient than using MediaPipe as well as a Kinect - there is a Kinect CHOP that will spit out all of your pose tracking info