Hello, I’ve been seeing this for a while and I was wondering if it would be possible to simulate something similar in TouchDesigner, with Mediapipe Pose and Face Tracking, or with a Depth Map:
Does anyone know if this has ever been done? Thank you very much.
I’m also curious about real-time tracking + Projection within TouchDesigner.
We saw this amazing example by Paul Lacroix about 10 years ago, but I haven’t seen anyone try tracking and projecting on people recently? If anyone has more recent examples and or breakdowns I would love to see them.
I saw that Paul used a Optitrack camera and wondered if any were compatible with TouchDesigner / or MediaPipe to run marker/markerless at ultra high frame rates? Cameras | Optitrack.com
I did something similar as a protoype a loooong time ago:
The basic idea is to calibrate a 3D camera and a projector in order to treat the projector as a camera. This works because a projector is, in fact, the inverse of a camera, and the goal is to render from the projector’s perspective.
Fast way:
This can be done manually, similar to the Camschnappr 3D model-to-projector calibration. The result works for static objects, but the calibration data is not very precise.
Best way:
First, you determine the intrinsics of the camera (FOV, lens distortion, etc.). Then, you calibrate the intrinsics of the projector with the help of the calibrated camera (using structured light / Gray code). In the final step, you calibrate the extrinsics (translation and rotation) between the 3D camera and the projector.
With this process, you obtain all the necessary data for the projector-as-camera setup (FOV, position, rotation, etc.).
Everything can be done inside TouchDesigner using OpenCV. Once calibrated, the whole setup works in real time.