Several issues after extensive Orbbec Femto Bolt testing

I spent the past few days extensively and obsessively testing the Orbbec Femto Bolt on the latest 2023.11340, and here’s what I found.

The more mundane findings:

  • Adding Orbbec TOP, deleting it, then adding a Kinect Azure TOP and going Orbbec mode crashes TD silently
  • Adding Orbbec/Azure TOP, deleting it, and adding again can cause the stream to become choppy/delayed: this is a somewhat documented issue

Now to the meat of it…Throughout my rigorous testing I’ve experienced the following:

  • Things generally work well with Orbbec TOPs :+1:
  • Things generally work well with a single Azure TOP

Here’s where the big issues start:

  • Azure CHOP is delayed if there’s an Azure TOP in Color mode; but not always
  • Having an Azure TOP and an Azure Select TOP:
    • Delayed Color and Depth (includes PointCloud) streams; talking >1 sec; but not always
    • Staggering/skipping Color and Depth (includes PointCloud) streams;
    • The worst offender consistently is Player Index: see this vid. This skipping action happens not only in combination with the Color stream but also other Depth streams!

Here are my specs:
Build: TD2023.11340
OS: Win11 Pro
CPU: AMD 7950x
GPU: Nvidia RTX4090 Gigabyte Gaming OC
MoBo: Gigabyte B650 Aorus Elite AX

I am (now) aware that the Orbbec/Azure is also unkeen of ASMedia USB controllers that a majority of AMD MoBos have, and I’ve already installed a Renesas/uPD720201 based PCIe USB expander for my Kinect v2 which is supposed to work with Azure.

Thinking it might be the cable/port, I tried different combinations, all of them similar results. I even updated the Orbbec firmware, which maybe improved things a tiny bit?

In this article they mention a PCIe USB controller card which they claim is tested and validated. Ordered it and just finished installing and testing after a firmware and driver (which I had to roll back cause the device just disappeared from TD) update of the card with no real improvement.

Wanting to get to the bottom of this I also asked a friend to test on his Intel + 4090 build (and even a different sensor device). Same results. Also tested on my Legion 7 Intel + 3080 laptop. Here I didn’t experience the skipping/staggering only the huge delay between the streams, but even more consistently.

With this it’s safe to say there are some grand issues with Orbbec/Azure, not having found any configuration that fully works reliably. Tomorrow I’m visiting some more friends to get even more data :slight_smile:

Can anyone else chime in? How is it in the Derivative office?

2 Likes

Thanks for the analysis.

I’m not surprised there are problems switching back and forth between the Orbbec and Kinect Azure TOP in the same project. Orbbec didn’t really design the Kinect emulator and the native SDK to be used together and I suspect there is host-side information that won’t be synced properly by the two systems which is likely leading to the crash.

Generally speaking, if you need the skeleton tracking features on the Orbbec (player index or CHOP), you should use all Kinect Azure TOPs/CHOPs. If you only need image data (depth, color or IR) then you should use the Orbbec TOP.

I will add some of this information about this to the docs.

As far as the delays are concerned, I’ve only worked with the Femto Mega but the performance has been pretty good on my development machines (which are both Intel). I do recall there was some significant lag on one system we tested with., but I don’t remember the solution right now. I will look into this further and get back to you.

Regarding sync issues between the streams: anything that relies on the body tracking is always going to be at least a few frames behind. The skeleton tracking is all done on the host machine GPU after the depth/color images are transferred from the camera. There is a ‘Sync’ parameter on the Kinect Azure TOP that will delay the color or depth images so that they match the current skeleton data.

I hope that’s somewhat useful to you. I’ll do some more testing with my machine here tomorrow and see if I can get you any further answers.

Hey Rob, thanks for looking into it!

Thank you for mentioning, however I was aware of not being able to mix the Orbbec and Azure TOPs/CHOP and did not refer to using them at the same time, after having read the docs.

The delay I am mentioning between the streams is much greater than a few frames, rather it’s a full second at least. As well as the worse issue of skipping/staggering. Can record later on both my Intel and AMD (with the depthkit-recommended USB controller) machines.

Sorry for the delayed response. I was having some issues as well when I went back to testing that turned out to just be an insufficient cable. At least with the Femto Mega camera, it’s important to have a cable that supports enough power and data otherwise you can get some odd behaviours (my camera wouldn’t even connect with usb c cable I was originally trying). In my testing a 3A + 10Gbs rated USB C to USB C seems to work well.

Regarding mixing ops: just to clarify it will still cause a problem if you delete an Orbbec TOP and then add a Kinect TOP afterwards (or vice versa). The Orbbec driver remains in memory as long as touch is open, so you would need to restart touch to properly reset things.

I’ve got an idea on how I might be able to keep the two systems in parallel which I hope to look into when I get a sec.

As far as the delay, I haven’t been able to reproduce that yet on my machine. At least with the femto mega and my nvidia 3070 card, the player index image is pretty closely sync’d to the color or depth images (any delay is pretty hard to notice). I’m curious if the Bolt behaves differently than the mega here - I’ll reach out to Orbbec to see if they have any thoughts.

Cheers @robmc, I suspect it will behave differently. Will try to bring my Bolt to some more friends to test. So far I’m quite convinced.

Hi @robmc sorry for pushing this, is there any update perhaps? Has anyone at derivative been able to test the Bolt?

This is a prettt pressing issue for me having budgeted the Bolt for multiple installations expecting the Azure performance.

Someone over my Patreon, discussing this issue has also said the following regarding the Femto Mega:

"I read over the forum post and plan on replying there soon, but with the Mega connected via USB C with the supplied cable and a .toe that’s simply an azure top showing RGB (limited to 1280x1280) linked to an azure select top showing player index, the index does not have latency but the RGB does occasionly encounter latency and stutter. Then if I add a 2nd azure select top set to point cloud then the point cloud and RGB show occasional latency while the player index is rock solid with no latency. I plan on testing a bit more with all other software shutdown and altering the power management plan. "

Hey, I messaged Orbbec again a few days ago and they just got back to me.

We haven’t tested with the Femto Bolt yet ourselves, so I’m just working on the information they’ve given me.

In their words: "… the expected performance from Femto Bolt should be exactly same as Femto Mega. They have the exact same depth module.

The main difference between each other is Mega has build-in processors(Jetson Nano) to process the depth info, however, Bolt must rely on the host to do that. If the host has limited performance, it may affect the depth results to produce different performance."

When you’re experiencing the lag or stutters, have you looked at the image errors in the Info CHOP? It’s not very specific, but if there are any problems retreiving the color image then that number would be increasing. The info chop also shows the timestamps for the tracking and color information which might at least quantify the problem.

Hi Rob,

I have been away from my studio and will be for a bit. I’ll do some testing then for sure!

So I recently got a Femto Mega for a project at work and turns out I am having the same latency issues as @function.store . When using an Azure TOP set to Color, one Azure select TOP set to Player index and a 2nd Azure select set to Depth the, Player index is lagging behind the other 2 significantly to the point where it might be unusable.

When testing this same setup with a Azure Kinect, I dont see these problems at all.

Ive tested this with 2 different machines, both with Intel CPU’s and NVIDIA GPU’s. In addition, i’m using the supplied cable from Orbbec in addition to the latest version of TD.

I will check the info CHOP tomorrow for any errors but any help or update on this would be helpful.

2 Likes

I’m experiencing the same issues with Femto Bolt, is unusable with the Azure Kinect Top + Azure Select Top, I need to compose the player index with color image and the color is delayed 1 second and with stuttering. “Sync image to body tracking” does not fix the issue, same setup with Kinect Azure works perfect, but with Femto Bolt it has lot of issues, I’ve tried it on 3 different windows 10 and 11 machines with RTX4090, are you planning to solve this? is it on your timeline? please! let me know if I can help with any testings, thanks! I’ve tested it on TD 2023.11340 and 11560.

Using Femto Bolt firmware v1.0.9 you’ll have the 1 second lag on the color image if you use the player index at the same time, but on latest v1.1.2 firmware it is IMPOSSIBLE to use the color and player index at the same time with the Kinect Azure TOPs, it will just freeze both images. Orbbec TOPs are working ok but they don’t support all the features that Kinect Azure TOPs do. Please @robmc fix Kinect Azure TOPs so they work ok with Femto Bolt or update Orbbec TOPs with all the features. Thanks

I have successfully used FemtoBolt with TouchDesigner, and the performance is consistent with Azure Kinect. However, the driver for FemtoBolt integrated into the latest version of TouchDesigner is outdated, causing some issues. Updating to the latest FemtoBolt driver file should resolve these problems.
ref:
FemtoBolt + TouchDesigner Quick Start
Femto Bolt+TouchDesigner Driver Update
https://www.youtube.com/watch?v=ipg982jHJoQ

2 Likes

I’ve updated the Orbbec SDK to the latest (1.10.8) internally and should appear in our next release.

2 Likes

thanks! this fixed the 1 second delay, and is compatible with latest Femto firmware 1.1.2, but I still have stuttering and choppy rgb image and depth image when Player Index is also used, please check the screen recording, what could be the issue that is causing this? The problem appears only when using Player Index, if not it runs perfect,
thanks!
2024-06-18 17-24-20-FemtoBolt.zip (3.9 MB)

1 Like

Same issue here, after updating the dll’s as @jackxu described, Orbbec with firmware 1.1.2 is working on TD 11560 but color playback of Azure TOP is choppy when Play Index and Color are used at the same time. Thanks for the fast support

If anybody wants to test out the new Orbbec SDK you can try this build here: Dropbox

The lag with the point cloud image should be better - Orbbec has implemented a new undistort method which works much faster than the old coordinate transforms.

I am still seeing occasional problems with the Kinect Azure TOP and the player index field which I’m looking into. If you turn on the TOUCH_TEXT_CONSOLE environment variable you can see Orbbec outputting a depth initialization error when the problem is happening, but it doesn’t give much information and happens inconsistently for me (I can sometimes toggle the camera active parameter and then it works fine)

The error seems to be happening in a call between the Microsoft body tracking library and the Orbbec Kinect emulation library. I’ve posted the issue to Orbbec and will post if I find out anything new.

1 Like

TD11839_FemtoBolt.zip (3.7 MB)
Hi @robmc, thanks for the update, I’ve tried it but I still have the same issue, when using color and player index, the color is choppy, loosing a lot frames each 4 or 7 seconds randomly. I’ve made a screen recording and saved the touch text console log in the attached zip. If you see the video, before adding the player index, color image of the Femto works great. I hope you can solve this soon, let me know if you need any additional testings, I’d glad to help. Thanks for the support.

I’m seeing the same issue as pogicAV with the Femto Mega in the 11839 build with no image errors in the info CHOP. Running on a 4090 / 7945HX with Win 11 23H2 and NVIDIA Studio Driver 555.99. Also, the color, depth, & bodytracking timestamps all show matching values.

Thanks for the feedback and the video - it’s very useful to see what kind of problems you’re experiencing.

So far I haven’t been able to reproduce those stutters with the color image on my Femto Mega camera running on a 3070 mobile card. As I mentioned above, occasionally I get errors that prevent it from starting, but once it is running things are pretty smooth.

In the past, I think similar issues with the original Kinect Azure tended to be problems with the USB connection. Have you guys tried different cables, or different ports on your machine?

Do you have similar stuttering if it is the color and depth images, or does it only happen with the player index? Does it happen with just the Color image and the Kinect Azure CHOP (to enable the body tracking, but without the player index)?

Good news, after trying everything, I could “solve” the problem, switching the Body Tracking Processing Mode from GPU Default to CPU fixed the issue! But now my CPU runs at 100% utilization (i9 13900k) only because of the body tracking, if I switch it to GPU it runs at 15% much better.
Seems to be some conflict with the GPU when using player index with gpu tracking and any other kinect image (Color or IR or Pointcloud or Depth). If not using player index but using Kinect Azure CHOP, there is no problem to simultanously use any other of the kinect azure images.

1 Like