Dynamic Projection Mapping - Track several different objects simultaneously


We are trying to map several virtual objects in a 3d scene to their physical counterparts in real space. The physical tokens are supposed to alter the projection (which is trying to emulate water with FLEX particles) by acting like obstacles. We have a kinect v2 tracking the scene.

So far we have managed to track reflective surfaces by their IR value as well as objects by their color. The solution of tracking by IR value does not scale very well to more objects since it seems hard to make good thresholding work well - we will be projection onto a physical space with different heights, making it harder to discriminate between what and what not to track. Tracking by color gets tricky since a projected color image will always be present in the physical scene, affecting the color value of the tokens.

Any idea how to solve this? We have started looking towards fiducial markers/qr codes. Is that something that works well in TD, considering we have to track several objects at once. How would one go about doing that?

Greatful for your input!


Depending on the scale of the area, Vive Trackers provide excellent tracking and accurate enough for projection mapping. Something like this although in this video they are using Vive controllers not the much smaller trackers (it was before the trackers were released).

There are also more expensive options like Blacktrax and Optitrack, but it really depends on the size of the space and the accuracy you need.

That seems to work really well. However, we want the tokens to represent different things - obstacles - in the physical environment. Using vive controllers would be a bit clumsy and not to mention we would have to have at least 6 of them. The area we will be tracking is approximately 1.5x1 meter, a tabletop.

I have been looking into reactivision or openCV for tracking fiducial markers. Is that something that could work? There seems to be very little tutorials on how to use openCV for touchdesigner.


Yes, tracking fiducial markers would certainly work.
For a primer on setting up openCV with TouchDesigner, start here: https://docs.derivative.ca/OpenCV

Github is also another treasure when looking to get started: https://github.com/search?q=touchdesigner+openCV or more generally this: https://github.com/search?q=touchdesigner

If you get stuck along the way post your questions in the forum they’ll be someone with some more openCV experience than I to help.

Hi Viceur

For opencv fiduciary tracking you can see an example on learnopencv.com in python and c++. Maybe it goes deeper what you want but it should be a good beginning.