Several issues after extensive Orbbec Femto Bolt testing

I spent the past few days extensively and obsessively testing the Orbbec Femto Bolt on the latest 2023.11340, and here’s what I found.

The more mundane findings:

  • Adding Orbbec TOP, deleting it, then adding a Kinect Azure TOP and going Orbbec mode crashes TD silently
  • Adding Orbbec/Azure TOP, deleting it, and adding again can cause the stream to become choppy/delayed: this is a somewhat documented issue

Now to the meat of it…Throughout my rigorous testing I’ve experienced the following:

  • Things generally work well with Orbbec TOPs :+1:
  • Things generally work well with a single Azure TOP

Here’s where the big issues start:

  • Azure CHOP is delayed if there’s an Azure TOP in Color mode; but not always
  • Having an Azure TOP and an Azure Select TOP:
    • Delayed Color and Depth (includes PointCloud) streams; talking >1 sec; but not always
    • Staggering/skipping Color and Depth (includes PointCloud) streams;
    • The worst offender consistently is Player Index: see this vid. This skipping action happens not only in combination with the Color stream but also other Depth streams!

Here are my specs:
Build: TD2023.11340
OS: Win11 Pro
CPU: AMD 7950x
GPU: Nvidia RTX4090 Gigabyte Gaming OC
MoBo: Gigabyte B650 Aorus Elite AX

I am (now) aware that the Orbbec/Azure is also unkeen of ASMedia USB controllers that a majority of AMD MoBos have, and I’ve already installed a Renesas/uPD720201 based PCIe USB expander for my Kinect v2 which is supposed to work with Azure.

Thinking it might be the cable/port, I tried different combinations, all of them similar results. I even updated the Orbbec firmware, which maybe improved things a tiny bit?

In this article they mention a PCIe USB controller card which they claim is tested and validated. Ordered it and just finished installing and testing after a firmware and driver (which I had to roll back cause the device just disappeared from TD) update of the card with no real improvement.

Wanting to get to the bottom of this I also asked a friend to test on his Intel + 4090 build (and even a different sensor device). Same results. Also tested on my Legion 7 Intel + 3080 laptop. Here I didn’t experience the skipping/staggering only the huge delay between the streams, but even more consistently.

With this it’s safe to say there are some grand issues with Orbbec/Azure, not having found any configuration that fully works reliably. Tomorrow I’m visiting some more friends to get even more data :slight_smile:

Can anyone else chime in? How is it in the Derivative office?

Thanks for the analysis.

I’m not surprised there are problems switching back and forth between the Orbbec and Kinect Azure TOP in the same project. Orbbec didn’t really design the Kinect emulator and the native SDK to be used together and I suspect there is host-side information that won’t be synced properly by the two systems which is likely leading to the crash.

Generally speaking, if you need the skeleton tracking features on the Orbbec (player index or CHOP), you should use all Kinect Azure TOPs/CHOPs. If you only need image data (depth, color or IR) then you should use the Orbbec TOP.

I will add some of this information about this to the docs.

As far as the delays are concerned, I’ve only worked with the Femto Mega but the performance has been pretty good on my development machines (which are both Intel). I do recall there was some significant lag on one system we tested with., but I don’t remember the solution right now. I will look into this further and get back to you.

Regarding sync issues between the streams: anything that relies on the body tracking is always going to be at least a few frames behind. The skeleton tracking is all done on the host machine GPU after the depth/color images are transferred from the camera. There is a ‘Sync’ parameter on the Kinect Azure TOP that will delay the color or depth images so that they match the current skeleton data.

I hope that’s somewhat useful to you. I’ll do some more testing with my machine here tomorrow and see if I can get you any further answers.

Hey Rob, thanks for looking into it!

Thank you for mentioning, however I was aware of not being able to mix the Orbbec and Azure TOPs/CHOP and did not refer to using them at the same time, after having read the docs.

The delay I am mentioning between the streams is much greater than a few frames, rather it’s a full second at least. As well as the worse issue of skipping/staggering. Can record later on both my Intel and AMD (with the depthkit-recommended USB controller) machines.

Sorry for the delayed response. I was having some issues as well when I went back to testing that turned out to just be an insufficient cable. At least with the Femto Mega camera, it’s important to have a cable that supports enough power and data otherwise you can get some odd behaviours (my camera wouldn’t even connect with usb c cable I was originally trying). In my testing a 3A + 10Gbs rated USB C to USB C seems to work well.

Regarding mixing ops: just to clarify it will still cause a problem if you delete an Orbbec TOP and then add a Kinect TOP afterwards (or vice versa). The Orbbec driver remains in memory as long as touch is open, so you would need to restart touch to properly reset things.

I’ve got an idea on how I might be able to keep the two systems in parallel which I hope to look into when I get a sec.

As far as the delay, I haven’t been able to reproduce that yet on my machine. At least with the femto mega and my nvidia 3070 card, the player index image is pretty closely sync’d to the color or depth images (any delay is pretty hard to notice). I’m curious if the Bolt behaves differently than the mega here - I’ll reach out to Orbbec to see if they have any thoughts.

Cheers @robmc, I suspect it will behave differently. Will try to bring my Bolt to some more friends to test. So far I’m quite convinced.

Hi @robmc sorry for pushing this, is there any update perhaps? Has anyone at derivative been able to test the Bolt?

This is a prettt pressing issue for me having budgeted the Bolt for multiple installations expecting the Azure performance.

Someone over my Patreon, discussing this issue has also said the following regarding the Femto Mega:

"I read over the forum post and plan on replying there soon, but with the Mega connected via USB C with the supplied cable and a .toe that’s simply an azure top showing RGB (limited to 1280x1280) linked to an azure select top showing player index, the index does not have latency but the RGB does occasionly encounter latency and stutter. Then if I add a 2nd azure select top set to point cloud then the point cloud and RGB show occasional latency while the player index is rock solid with no latency. I plan on testing a bit more with all other software shutdown and altering the power management plan. "