Hey everyone. I’m having trouble with my player index on the Kinect azure. It’s really choppy. So it’s making it hard to mask things out with it. My computer is strong enough so I know that’s not the problem. The Kinect V2 seems to be way smoother. Can anyone give any input?
1 Like
Also experiencing the same issues as the original poster. The Kinect SDK runs way smoother than in Touchdesigner. Has anybody found a workaround?
Thanks
I think you need to be a little more explicit when you say ‘choppy’'. I assume you are referring to visual appearance and not playback performance.
So my very simplistic understanding of what is happening.
The Azure Kinect uses AI generated depth segmentation data to create the Player Index image. Its pretty useless for any sort of masking.
It is possible to use the depth map with a lot of fiddling. Malcolm posted a file I believe elsewhere on the forums that should help, do a search.
However I’ve had the best results using PointCloud data to extract a depth mask.
Hello yes I’m talking about using the player index as a mask because with the V2 it was easy to do so. I’m using the azure to get a pointcloud but I want to have only me and not everything else in my space. What I mean by choppy is that is seem to not give me a clean silhouette. Parts of my body go in and out. So you said this possible with the depth map? I will try to find that post you told me about. Thanks for reply.
Just to follow up: when you say the Kinect SDK runs smoother, are you still comparing the Kinect Azure to the old version 2, or are you finding better results in the Azure SDK viewer (Azure Kinect Body Tracking Viewer or Azure Kinect Viewer 1.4) compared to the TouchDesigner version?
Thanks for the follow up -
Comparing Azure Kinect Body Tracking Viewer to viewing player index in touchdesigner.
What I do notice is not the visual appearance discrepancy (the visual appearance matches the shapes seen in Body Tracking Viewer, with holes and such), however I have not experienced frozen frames while viewing Body Tracking Viewer as I have in TD, where at times I experience 2-3 second frozen frames
Thanks for the details. We weren’t aware of any freezes like you’re describing.
Are you using the latest version of Touch Designer (2020.23680 that was released on May 20)? There were some internal changes in that build that were supposed to improve how data was uploaded to the GPU that could have affected things.
When you say it freezes for 2-3 seconds, is that just the Kinect Azure node or all of TouchDesigner? Is it only the body-tracking data that freezes or do you have problems with the color camera images as well? Are you using the ‘Sync Images to Body Tracking’ feature?
Thank you for your reply robmc
I will check everything that you have mentioned and report back
Hello,
I’m also experiencing this issue. I have two Kinect Azure Devices and checked them both on 3 different high-end PC’s (i7, i9, RTX 3090 and such).
What I experience is that Azure Kinect feeds start to get choppy (stutter, drop frames) whenever I use Player Index feed.
The CPU is speeding up to 99-100% utilisation on all cores (checked in Task Manager) in a couple of seconds. There is most definitely some kind of bug there.
It is such even in the clean project where I just use Azure Kinect and Azure Kinect Select TOPs.
Even on it’s own, the Player Index stream in Azure Kinect TOP is about eats about 85-90% CPU performance.
I use the newst version of TD with commercial license.
I suspect it’s the body tracking system that is consuming that much CPU and causing problems. After the Kinect update last year, Microsoft switched the azure to use DirectML rather than CUDA as the default which seems to have notably higher CPU usuage.
We’re actually testing out an experimental update that lets you select the AI engine to run the body tracking on which may work better on your 3090 card. I’ve still got a few tests to run on it here, but I will post the link when its available and you can see if that helps.
Separately, the kinect does consume a large amount of USB bandwidth which could cause stuttering - especially with multiple devices attached. Microsoft has more information here: Azure Kinect known issues and troubleshooting | Microsoft Learn
2 Likes
Here’s the experimental Kinect build if you want to try it. In the Kinect Azure properties, you’ll notice a new ‘Body Tracking Processing Mode’ option. If you switch this to CUDA, you should see a significant drop in CPU usage which will hopefully help some of the performance issues you have.
https://www.dropbox.com/s/uebp2ci42vfmykh/TouchDesigner.2021.16285.exe?dl=0
Note, this version uses a new installer and has some known conflicts with some of the Nvidia TOPs, so let us know if you run into any problems.
2 Likes
Thanks a lot!
I’ll check it out and report 
2 Likes
Just out of curiosity: Did switching to CUDA change the performance? For me it definitely is way smoother than DirectML, but I still get some stuttering every now and then. Also I don’t see much difference between CUDA and TensorRT, running on a AMD Ryzen 5600x and RTX 3080 with 32GB RAM
This CUDA option is much better, thanks for adding this @robmc ! I assume this will make it into a 2022 release at somepoint, very exciting.
I do often worry that MS have abandoned the Azure, would be so good to get its body tracking and player index at least on par with Kinect v2, such little movement from them 
1 Like
All of the changes are in the new 2022 official release we just launched this week.
I’m not too sure on what the long term plans are for the Kinect Azure. There’s some talk about a more general depth platform (Azure Depth Platform - Azure Depth Platform) and some hints that they might be adding 3rd party hardware eventually in this thread: What is the status of this SDK? · Issue #1496 · microsoft/Azure-Kinect-Sensor-SDK · GitHub
1 Like