Mock live feed into Kinect Azure TOP

In TouchDesigner InSession - June 25th 2021

a neat trick is shown for recording and playing back Kinect v2 data in one of the SDK tools and then playing it back so TouchDesigner treats this as a live feed.

Has anyone got an equivalent system working for Azure Kinect ?

The AZK SDK comes with a CLI k4arecorder.exe tool that records mkv files with all the data in various tracks and a k4aviewer.exe that plays them back. However the TOP isn’t tricked into treating playback as a live source.

Unfortunately the Kinect Azure SDK doesn’t support the same feature as the v2 kinect for faking a live stream with a the viewer app.

For the azure, you need to manually stream the data from the mkv file and TouchDesigner doesn’t currently support that. However, you can get similar results using a sequence of EXR files generated by the Movie File Out and loaded by the Point File In. Let me know if you have any questions about that.

EDIT: I should add that if you’ve got an existing mkv file, I believe you can extract an EXR sequence from that using the ffmpeg tool.

1 Like

Hey @michela

You can also use (with partial success) mkvtoolnix to extract the individual tracks. It’s a quite complete suite of tools related to mkv files.

You’ll find MKVtoolnix here: MKVToolNix download latest version
The documentation is here: MKVToolNix Documentation – Matroska tools for Linux/Unix and Windows

If you are not into CLI, you can also use gMKVExtractGUI: gMKVExtractGUI v2.6.4 - Doom9's Forum

Using gMKVExtractGUI, you’ll have to reference the install folder of mkvtoolnix (first field at the top), you can then drag n drop the Kinect Azure recording in “Input files” and it should give you all the details.

Note that it doesn’t like the IMU data.

You can also maybe have a look at the extra data that you can get from Cue_sheet and everything else from the MKV, not sure exactly how Microsoft is packing everything but here is some extra doc: Use Azure Kinect Sensor SDK to record file format | Microsoft Learn.

I was able to load those in TD after extracting, using Microsoft samples recordings:

Edit: Azure Samples: https://www.microsoft.com/en-us/download/details.aspx?id=58385&WT.mc_id=
Edit 2: ffmpeg or mkvtoolnix are both suggested in the documentation from Microsoft :slight_smile:

Best,
Michel

2 Likes