School Project: Interactive Motion Tracking Projection

General TouchDesigner Discussion
1

Hi all,
I’m struggling as beginner to find a way to create an interactive projection. Basically I’m trying to create a motion tracking experience through regular webcam where students can manipulate through gestures their own drawing projected on a wall.
I know of a particular art project where children scan their drawing and are displayed on wall in real-time for them to interact with.
Any tips or examples of how to use scanned drawing in touchdesigner and plugins / components i can use maybe would be very welcomed.
Thank you

2

Hi @Atelierista,

i see two parts to it:

  • familiarize yourself with the MediaPipe component by @tblankensmith to use a webcam for interactions
  • define what the interactions should look like: for example are the images moving on the screen and avoid the kid’s hands? Or when touching a drawing does it change it’s color?

cheers
Markus

3

Hi Markus and thanks for your reply,
I have been learning about MediaPipe, now I am thinking how to add images which will move avoiding children’s body, with full body tracking. That seems to be to part I am stuck at.

Best,
Cata

4

Hi @Atelierista,

a rather simplistic approach could be to use particles and the position of anyone on camera as a negative force to make the particles “avoid” that area.
Have a look at the OP Snippets of the Particle SOP or online tutorials on the particlesGPU in the palette.

Otherwise you might be looking for boid like behavior which would include drawings avoiding each other as well as the areas people occupy on camera. This is usually a bit more involved and will need glsl knowledge at the moment to complete. A code free approach that might be adaptable can be found on @paketa12’s patreon:

Hope this helps
Markus

5

hi Markus any idea how to connect MediaPipe to OP Snippets?
thanks.

6

Hi @Atelierista,

usually you would copy an example network from OPSnippets and then it depends what aspects of the network you want to control. If you can share your progress so far it could be easier to give specific advise.

cheers
Markus

7

Thank you, I managed to do a hand movement reaction with particlesGPU and MediaPipe, now I’d like to know how to enable reaction from several hands (10 individuals) as it seems to only react to one hand at a time. Any suggestion?
Thank you

8

Hi @Atelierista,

MediaPipe should track both hands, I’m not sure how many people it can track - but this is essentially now a task for you: find out what software or hardware is able to track 10 hands at once?

cheers
Markus