Hello, i would like to know if is expected in future to have orbbec chop for skeleton tracking, in case we have to continue to use Azure for that would be great to have on azure kinet TOP the possibilities to connect to the camera using ethernet cable and IP (is far better for the cabling instead of the usb C connector), and other question about the Femto is when is schedued to have control over the color settings of the camera natively in touchdesigner including the Sync option, right now i didnāt understand if once the camera is set to slave and master in the SDK after work in sync also in TD.
3 Likes
There are no plans for an Orbbec CHOP at the moment ā¦ the Femto body tracking features all still rely on the Kinect Azure libraries and it is my understanding that Microsoft still controls these. The Femto body tracking works right now by having the Orbbec camera emulate the Kinect Azure so that the body tracking still thinks its talking to a Microsoft camera. This only works by limiting the Orbbec camera to features that are supported by the Microsoft ones, so I think it still makes sense to run this through the Kinect Azure TOP.
I believe Orbbec did have their own native body tracking application in their Astra specific SDK, but that was deprecated a few years ago and there are no plans to implement that in TouchDesigner.
As far as getting ethernet working for the Femto Mega with body tracking: it might be possible, but there are a few things that need to be implemented first. Since body tracking requires using the Kinect API which doesnāt have ethernet, there is no way to give an IP address. Orbbec may have worked around this in their latest SDKs with the enabling of an auto IP detection system, but we havenāt had a chance to test this yet.
The other obstacle is the color camera encoding. Since the ethernet connection has lower bandwith than USB, it only supports H264 encoding which is not currently supported in the Kinect or Orbbec TOPs. We have some work going on internally to generalize our H264 decoding but that isnāt quite ready yet.
As far as color and sync options on the native Orbbec TOP, I recommend still using the Kinect Azure TOP for that at the moment. Weād like to add some of those features for the general Orbbec cameras, but I donāt have a timeline for that right now.
Hope that answers your questions. Let me know if I can do anything else.
Hi @robmc thanks for the info on the bolt.
Iām using the latest build 2023 11600 with a Femto bolt and Iām experiencing some lag/delay, probably about 1second on the depth image compared to the color image. Is this to be expected, and are there any tips you can share which may help reduce the IR lag.
Thanks
Andy
1 Like
Iām also experiencing this on the 512*512 image and above, only the NFOV views are full FPS and low latency.
On the current version 11600 as well!
@robmc
With the latest build 2023.11600 weāre experiencing significant delay and lag when streaming PointCloud from the Orbbec TOP. Weāve been testing with USB and Ethernet and we have the same results, the data is lagging 1-2 seconds behind. This is not a problem when using the Depth image instead of PointCloud.
With build 2023.11340 this is not an issue!
Everything runs smoothly when using OrbbecViewer
Using the Kinect Azure TOP isnāt really a possibility as we intend to use the PoE feature of the Orbbec and are pointing at their IP address. Any possible fix about this?
Thanks for the report. So far I havenāt been able to reproduce the lag issues here, but Iāve been planning to spend some more time on this shortly to try and figure out what is going on.
The fact that the behaviour changed between versions of TouchDesigner is interesting. The main difference between those builds is the upgrade to version 1.9 of the Orbbec SDK and the addition of Orbbecās undistort function on the point cloud data.
I will let you know what I find out.
Also, another user recently found a way of using the Orbbec PoE feature with the Kinect Azure TOP. If you add an orbbec config file alongside your toe file with the āEnumerateNetDevicesā flag turned on, then any connected ethernet cameras should appear in the device list like the USB cameras. Orbbec has some more information on the config file here: K4A Access Femto Mega Network Mode - ORBBEC - 3D Vision for a 3D World
3 Likes
@robmc This is an interesting development. Is there documentation or a write up on this Userās findings that you could point me towards? Or are you able to tag them in this thread? Iām assuming that this means that the Femto Mega can be used with the Kinect Azure TOPs / CHOPs to do both depth camera stuff as well as skeletal tracking over ethernet?
@k.anti Are you able to test the process that Rob described above and report your findings?
I still havenāt been able to try it myself yet, but most of the details are in that link I gave to the Orbbec website. Your Femto Mega camera needs the right firmware and the config file is included in the Orbbec SDK. The link also shows how to edit the config file to turn on network devices. The config file needs to be in the same folder as your toe file (or on the desktop if you start Touch without a toe file) for the Orbbec library to find it.
With this setup, you should be able to get the depth camera and body tracking data over ethernet using the Kinect Azure ops. If you only need the depth image, you can do that with the existing Orbbec TOP without any extra config file. Color images are not currently supported over ethernet.
Let me know if you have any problems with this. If I get a chance to test it out here I will update the documentation with more detail.
I have been succesful with this method with 6 Femto Mega in PoE using the config file along the .toe There is an abnormally long loading time to initialize the Kinect Azure TOP (around 1 minute) at first. It does then let you see the Femto as USB inputs and allows capture of the PointCloud at full framerate which we are still not able to do with the Orbbec TOP in version 2023.11600
We havenāt tested the skeletal tracking using this method yet!
1 Like
I am struggling to achieve this with two cameras to get a synced point cloudā¦Are you using a sync hub by orbbec? Which config file and .toe are you referring to (I assume the k4a over ethernet config but I donāt see a toe file on that page)?
Any help would be much appreciated!
I am not syncing them at the moment, our use case might not require it. Sorry for the confusion, the config file I mention is the k4a yes, the .toe file would be your TouchDesigner project file living beside like attached image.
@robmc Update to the stability of that method - the Kinect Azure TOPs are freezing altogether after some time. The project keeps running, but the PointCloud doesnāt get updated anymore.
Thank you for the clarification! Will try this and see if the point cloud freezing replicates on my end.
1 Like
Jumping in on this thread. Iām also seeing that the Orbbec TOP in 2023.11600 is running at a slower framerate like @hugoda was mentioning. This isnāt the case in 2023.11510, but in that version the Orbbec pointcloud is warped on my end. Iāve tried having the .xml file next to my .toe file as well and Iāve got it working so far with no stopping.
A feed freezing would be an issue. Does the TOP throw an error when it freezes? Are you able to unfreeze it by toggling āActiveā Off/On or does it require a network reboot?
Hi, following up on this a little as Iāve been doing more tests. First of all using the .xml file next to my .toe hasnāt been as stable as I had hoped depending on what computer I try it out on. Sometimes I just get a crash, itās behaving pretty randomly on my end and I havenāt been able to figure out why.
A work around Iāve been using is using the TD build 2023.11510 where there is usually the warping issue (addressed in later builds) but no frame rate issues. But when activating the depth align mode there is no more warping issue so it becomes a lot more usable in this build, although you will be using a bigger texture (that you can downscale if you want).
Iāve noticed in the more recent versions of Orbbec viewer (with more recent SDKs) that there is a framerate parameter for the depth mode that is by default at 15 fps in Orbbec viewer. I was wondering if that could be what is happening in TouchDesigner that itās getting that default value in the more recent builds. Just a thought.
Thanks for the ongoing feedback. Iāve been testing a new Orbbec SDK (1.10.8) today and unfortunately that lag with the point cloud seems to still be present. The lag is caused by Orbbecās undistort function which seems consistent with Orbbecās documentation that says: āThis function uses undistortion, which may result in longer processing time.ā
The Kinect Azure TOP uses a different undistortion system, so Iām looking into whether we can use that instead.
is this the same issue that is causing Femto Bolt to fail when using player index + color with the Kinect Azure TOPs?
I donāt think this would have any connection to those issues described with the Femto Bolt. The orbbec undistort functions that seems to be causing the lag here are not used in the Kinect Azure TOP.
Unfortunately, we donāt have one of the Bolt cameras and I havenāt been able to reproduce any of those issues with the Femto Mega so Iām not sure what is happening there.
Hopefully the new SDK might have some fixes for the problem.