Multiple RealSense inputs

Has anyone successfully used 3 RealSense cameras on the same Windows machine? All 3 would be of the same model.
Are there any limitations to keep in mind?

1 Like

Did you have any luck with this? I have an upcoming project in which I will probably have to use 4 RealSense cameras…

I haven’t yet gotten the hardware, but I asked @malcolm and he confirmed that it should work, though it might require additional USB PCI cards to handle the bandwidth.

1 Like

Was this successful? Can we use multiple Reaslsense cameras with Touchdesigner?

We ended up not being able to use it due to wiring and space limitations (e.g. USB-C can’t easily be 50+ ft long).
Instead at @malcolm’s suggestion we switched to using several of these:

How are you working with these? With the Python TOP? I’ve been looking at how to pull the video feeds or depth maps from the cameras, but to no avail.

I tried running them from within TD but it ended up being easier to have an external process that sends data to TD using OSC. The depthai_demo.py script has options for setting up callback functions and you can wire those up to send out osc. It has some opencv-based preview windows that can show object matches and other tracking info.

But I haven’t tried to get the actual image stream into TD. For that you might need to use their C++ API to copy the data into a TOP.

I was hoping they would also show up as regular camera input devices, but they don’t seem to, which sort of makes sense since it’s more of a separate computer than a standard input device.

2 Likes

Ah, gotcha, thank you! I picked up one of their ESP32 models (OAK-D-IOT-40), and am thinking of trying to set it up as a generic IP camera over wifi and streaming video in that way, rather than messing with SPI or anything.

It’s suggested at having a high chance of working here in a Github issue, so hopefully it will… BW1092: DepthAI ESP32 Reference Design | Embedded DepthAI Reference Design · Issue #10 · luxonis/depthai-hardware · GitHub

Hey tekt. This sounds amazing. Do you have any further documentation how you connected them together and got the data from multiple devices? I’d love to see how you all did it or even if you have a series of links or a pathway you could send me down to also achieve this. Much appreciated. Peter :slight_smile:

I used an external Python process that:

  1. Connects to whatever devices are available on the network and/or usb
  2. Sets up pipelines using spatial object tracking via mobilenet-ssd, with a script node in the pipeline that does some update filtering since the data can be a bit noisy
  3. Periodically (~10ms) polls them for tracking updates
  4. Sends tracking data and other status messages over OSC to TD
  5. Checks for timed out device connections and searches for replacement devices to attach

Then TD maintains a collection of tracked objects for each device, adding/updating/removing them when it receives messages, and writing their state including a device number to a DAT when things change.

Things got a bit more complex when dealing with fault tolerance. It may be less of an issue for USB, but for the ethernet ones if there’s a network disruption some of the API calls in the external process will stall. So you need to be careful about timeouts and checking for availability before requesting data.

https://docs.luxonis.com/projects/api/en/latest/tutorials/multiple/
https://docs.luxonis.com/projects/api/en/latest/samples/ObjectTracker/spatial_object_tracker/

I may be able to make some of the code available soonish, but the project containing it is proprietary.

This is amazing information. Thank you Tommy. And these cameras look super cool and accessible. I’m looking forward to getting stuck in with them :grinning:

1 Like

Hi, i was looking for connect multiple realsense camera and i found this post, the Luxonis camera look really interesting, do you think could be used for an interactive floor projection project?

Yeah it should work. It may make sense to wait for built-in support in TD. The other option is to use python to create a secondary process that sends data back to TD. That’s the approach I ended up using. But it sounds like native integration will be available in the not too distant future.

1 Like

Yea i have see that they are working on it