Help tracking people movement for Interactive floor

General TouchDesigner Discussion
,
1

Hi everyone,

I’m working an interactive floor installation where visuals are projected on the ground and follow/react to multiple people walking on it. Space setup will be a 6×5meter area, dim or low light conditions and a camera placed at ~2.5m height to capture the movement.

My issue is how to get people’s positions. The idea is either getting 3D coordinates through skeleton, point clouds, or using optical flow, thereby avoiding tracking consistency altogether.

I’m considering several approaches and I’d really appreciate some tips and opinions:

Option 1: Zed 2i / Femto bolt with skeletal tracking
Zed has 10m depth range, Femto ~6m. Both CHOPs should work out of the box(?). My only doubt is accuracy drops in dim/low light conditions for the Zed. (though I could pump visuals brightness and light the floor)

2: RGB/(IR?) Camera with MediaPipe
Is it viable to obtain multiple people’s coordinates in space? I couldn’t manage to find a specific guide. also infrared cameras would be better for low light conditions.

3: Depth camera to Point cloud
Don’t know if messing with shadow artifacts would complicate things. I did read in a previous post about merging multiple point clouds to fill occluded areas.

4: Lidar
Slamtec R1 placed at ankle height. but same issue as above, should use 2 of them to fill occlusions.

5: Bail and use Optical flow/blob tracking
Easier(?) solution, could avoid the issue of positions consistency. For optimal results the ideal positioning of the camera would be perpendicular to the floor, capturing a top-down view of the scene. I’d leave this option as last resort.

Which method did you try or would you suggest me to focus on? I appreciate any insights, experiences, or tips.

Sorry for the lenght of the post, this project is pushing me out of my knowledge zone so I’m overthinking it a bit.

Thanks in advance for your time!

2

Believe it or not, I’m actually a big fan of using Blob Tracking for interactive floors! It’s not a perfect solution, but depending on what kind of equipment you’re working with, it can be a fantastic tool.

For the last floor I made, I was working with an old, problematic Kinect V1 that was super finicky on actually capturing depth… it may as well have just been a camera! This forced me to get a bit creative with how I would accomplish everything. I found that it worked perfectly well to run the footage through a cache & blob tracking, messing with the settings, and refining the xy positions of those blobs to numbers that I could work with easier. Rendering those coordinates into the positions for clusters of geometry did the trick! From there, it’s just making sure you add some behind-the-scenes effects to refine the shape. It was a fun time figuring that all out!

If you have more of a budget to work with, or access to good equipment, point clouds are fun too! I tend to use those for interactions controlled by overlap instead of tracking, but it can for sure be done. Once you’re able to track those movements, the tricky part is dealing with light levels and particulates!

In my opinion, this all comes down to whichever method can deliver the consistent math you’re looking for. Once the numbers are in the palm of your hand, your options are infinite and you can do anything with them. I say you should just start with whichever method makes sense to you and can get you the values you’re looking for in a way you want to have them.

I hope this helps :slight_smile:

3

Hey! sorry for my late reply.
Thanks a lot! guess I’ll try first with my Kinect v2 using your blob tracking suggestion! I experimented a bit with a Realsense’ point cloud before realizing it has a range of 3meters :sweat_smile:.

Thanks again for your tips!