Orbbec Femto Mega

Hello, i would like to know if is expected in future to have orbbec chop for skeleton tracking, in case we have to continue to use Azure for that would be great to have on azure kinet TOP the possibilities to connect to the camera using ethernet cable and IP (is far better for the cabling instead of the usb C connector), and other question about the Femto is when is schedued to have control over the color settings of the camera natively in touchdesigner including the Sync option, right now i didn’t understand if once the camera is set to slave and master in the SDK after work in sync also in TD.

There are no plans for an Orbbec CHOP at the moment … the Femto body tracking features all still rely on the Kinect Azure libraries and it is my understanding that Microsoft still controls these. The Femto body tracking works right now by having the Orbbec camera emulate the Kinect Azure so that the body tracking still thinks its talking to a Microsoft camera. This only works by limiting the Orbbec camera to features that are supported by the Microsoft ones, so I think it still makes sense to run this through the Kinect Azure TOP.

I believe Orbbec did have their own native body tracking application in their Astra specific SDK, but that was deprecated a few years ago and there are no plans to implement that in TouchDesigner.

As far as getting ethernet working for the Femto Mega with body tracking: it might be possible, but there are a few things that need to be implemented first. Since body tracking requires using the Kinect API which doesn’t have ethernet, there is no way to give an IP address. Orbbec may have worked around this in their latest SDKs with the enabling of an auto IP detection system, but we haven’t had a chance to test this yet.

The other obstacle is the color camera encoding. Since the ethernet connection has lower bandwith than USB, it only supports H264 encoding which is not currently supported in the Kinect or Orbbec TOPs. We have some work going on internally to generalize our H264 decoding but that isn’t quite ready yet.

As far as color and sync options on the native Orbbec TOP, I recommend still using the Kinect Azure TOP for that at the moment. We’d like to add some of those features for the general Orbbec cameras, but I don’t have a timeline for that right now.

Hope that answers your questions. Let me know if I can do anything else.

Hi @robmc thanks for the info on the bolt.
I’m using the latest build 2023 11600 with a Femto bolt and I’m experiencing some lag/delay, probably about 1second on the depth image compared to the color image. Is this to be expected, and are there any tips you can share which may help reduce the IR lag.

Thanks

Andy

1 Like

I’m also experiencing this on the 512*512 image and above, only the NFOV views are full FPS and low latency.

On the current version 11600 as well!

@robmc
With the latest build 2023.11600 we’re experiencing significant delay and lag when streaming PointCloud from the Orbbec TOP. We’ve been testing with USB and Ethernet and we have the same results, the data is lagging 1-2 seconds behind. This is not a problem when using the Depth image instead of PointCloud.

With build 2023.11340 this is not an issue!

Everything runs smoothly when using OrbbecViewer

Using the Kinect Azure TOP isn’t really a possibility as we intend to use the PoE feature of the Orbbec and are pointing at their IP address. Any possible fix about this?

Thanks for the report. So far I haven’t been able to reproduce the lag issues here, but I’ve been planning to spend some more time on this shortly to try and figure out what is going on.

The fact that the behaviour changed between versions of TouchDesigner is interesting. The main difference between those builds is the upgrade to version 1.9 of the Orbbec SDK and the addition of Orbbec’s undistort function on the point cloud data.

I will let you know what I find out.

Also, another user recently found a way of using the Orbbec PoE feature with the Kinect Azure TOP. If you add an orbbec config file alongside your toe file with the ‘EnumerateNetDevices’ flag turned on, then any connected ethernet cameras should appear in the device list like the USB cameras. Orbbec has some more information on the config file here: K4A Access Femto Mega Network Mode - ORBBEC - 3D Vision for a 3D World

2 Likes