I’ve realised that the Azure’s user mask is not as good at the old Kinect V2, it can be quite patchy and my limbs tend to vanish.
So I’ve been trying to do background removal via GLSL and the depth map using a simple threshold, it works to a point but I’m having trouble removing the floor more than anything else.
left two image: azure mask | right two images: glsl mask
GLSL >>
...
if (color.r <= 0) {
color.a = 0.0;
}
if (color.r > uThreshold) {
color = vec4(0);
}
...
Are there good techniques out there? Perhaps one that combine the Azure mask and GLSL? Without painting my room green preferrably. Happy to hear any ideas!
I’ve been working away at this, here are two alternative methods I’ve come up with that work reasonably well. Less missing limbs anyway. Both operate on the azure point cloud feed. Hopefully they’re useful to someone!
Background Subraction
Computes a mask based on the difference between the live feed and a static capture. Stand out of shot, click ‘capture’ then you can cull further using depth, bottom, left, and right thresholds. azureBGSubtraction.tox (1.7 KB)
Body Bounding Box
Computes the min and max skeleton joint vectors and then culls any points outside of this bounding box. There are additional parameters to extend the box where needed (above the head for example). azureBGBoundingBox.tox (3.0 KB)
I’m also going to compile David Braun’s PyTorchTOP and run that over the colour image, see what results I get
Im interested in seeing your results over the colour image!
Besides background subtraction based on comparing to static “clean plate” input, I can think of one labor intensive method that could be useful for depth culling -
If you have a Kinect which is fixed and does not move, you can try modeling some 3d geo for the room (walls and floor), and match a touchdesigner camera angle to Kinect, then rendering a depth pass out of touchdesigner. This rendered depth pass can be used to cull out the depth info from Kinect.
I have another technik (cannot post it at the moment) with a capture of the empty stage (via Cache Top) and Glsl top to put alpha to zero when the pixel doenst move from the recorded value. I try to post it later. Its what I use here but for a real volume, not just a cache, So my VP can at different place from kinect.
That is an excellent idea!! I’ll give that a shot and let you know how I get on.
I did manage to compile the PyTorchGPU TOP but sadly it just kept crashing TD. Likely some dependency mismatch but reinstalling different versions of cuda and recompiling opencv takes hours so I gave up
