Hi everyone, maybe this is more a question for developers though.
How does the function “align/sync to color camera” work? The one in the Kinect Azure TOP, or the Azure Select TOP, whichever.
If the color camera is 1280x720, and the Depth map is 640x576ish, stretching the values with a Fit TOP doesn’t work.
Lens Distort TOP isnt helping much.
Stoner tool to make a UV map sounds like a horrendous idea.
It seems a bit of the depth data around the edges gets cut off when upscaled.
TD is really awesome that it has these little toggles available for both Kinect and Orbbec cameras from the viewers to make point cloud rendering super simple and fast, but I cant find anything in either Azure-SDK or the ORBBEC-Azure-Wrapper about how to align these two channels. The Azure SDK has a k4arecorder.exe program to save both the color and depth data out to a custom .mkv file but the channels apparently have different resolution still, and no arguments to align them when running the program. So is there like a public UV map or something they use that I cant seem to find?
Can anyone please share the secrets with me? I promise to use them for good!
Unfortunately I don’t have an exact breakdown of the math that is used to remap the images between cameras on the Kinect/Orbbec devices. Those toggles that you mentioned just trigger built-in functions from the Kinect SDK and TouchDesigner just receives the output.
Thank you so much!
Sorry for the late response (I didn’t get any notification), I’ll have a look at the colorCameraIntrinsics() and depthCameraIntrinsics() functions later this week and see if I can get the same looking UV remap.
Basically I’m setting up a multi camera point cloud video recording space. Unfortunately I can’t do it with touchdesigner because it basically slows down to 1fps per this thread but the Kinect SDK at least can export a .mkv with both the color and depth values, but theyre in different resolutions (idk why theres no argument to align them for the mkv recording). So when the recording is loaded into another program I cant figure out how to align them.
I hope that the camera intrinsics is not something that is updated too frequently in real time, hopefully a snapshot of the intrinsics from TD right before recording with the SDK will be a good enough lens-distorted UV map to at least align most of the points with their correct colors. I’ll see!
Ah, thanks for the explanation and sorry no one responded to that earlier thread. Unfortunately recording large lossless point cloud streams is something that remains challenging.
As you noted, I think the Kinect recorder does it by just saving the low bandwidth unprocessed depth & color streams directly to the mkv file and then relies on doing any projections/transformations when it is loaded.
The camera intrinsics for the Kinect should be static. My understanding is that they are set in the camera during calibration at the factory.
Thank you so much for the quick response, actually yes I’m giving it a go now, and I was hoping you could help me demystify a couple of things.
Intrinsics returns: cx, cy, fx, fy, k1, k2, k3, k4, k5, k6, codx, cody, p2, p1
But, the Lens Distort TOP has only k1-3, cx, cy, fx, fy
(im wondering how to use the 4,5,6 and what the codx cody are, they dont seem to be units and if they are, maybe im getting the relative/absolute pixels setting wrong).
Should I be… extracting focal point from the depth intrinsics or something like that to center it? Should I be… using a matrix from the getRemapTransform(src, target) function instead in some other operator?
Should I be… skipping all this and plugging the values into the Brown-Conrady model used in OpenCV?
Im a noob at this stuff btw. Thanks for the assistance in advance!
It doesn’t sounds like codx and cody are actually used, so you can likely ignore them.
As far as the higher K values, the Lens Distort TOP doesn’t use those because they don’t really have a noticeable impact on the distortion at the level we’re working with. Generally speaking, each higher K value has less impact on the output image with most of the effect at the very edges of the image. There is a lot of info here on the wiki page if you’re interested: Distortion (optics) - Wikipedia
If you’re comfortable with OpenCV, you should definitely be able to use that, but it should also be possible to do everything in touch with TOPs or GLSL if you’re comfortable with that.
It’s been a while since I’ve played with this stuff myself and I don’t have a Kinect camera hooked up at the moment, but I believe the steps would be to use the intrinsics to rectify the images first and then use the extrinsics (remap) transform to map between them.
Im really trying to get this with OpenCV in a script top but I just cannot figure it out after hours of trying and failing and numpy errors.
I’m getting the depth camera intrinsics. Im getting the color camera intrinsics. Doing just this, I was able to at least get a 1280x720 image with the point cloud inside of it, and lens distored to some degree, translated up and to the right, and much smaller than expected but then i learned I need the transformation matrix to stretch it correctly.
So now im getting the Homogeneous Transformation Matrix from Depth to Color.
I cannot understand how the cv.remap function works. I think i need to flatten the point cloud and add some homogeneous coordinates (according to chatgpt) but i dont understand why. I dont understand how to apply the transformation matrix, I think i need to project the ‘transformed points’ back onto a plane, I’m really just so lost. Can I please have some help?
My red output is backwards, its not at 1280x720 resolution, these numpy operations are bringing me down to 3fps, its just a mess.
Even if its not running every second, if I could just generate the one UV map from these intrinsics/matrix and plug point cloud + uv map into a remap TOP and get the output at 1280x720 aligned with color, I’d be happy.
Gave up trying with opencv and attempted in GLSL but no luck. I think a similar problem, nan values, probably when the pixel values are less than or equal to 0, but it could also be maybe Im using normalized UV coordinates (using 0->1, instead of like, from 0->1280?)
Im also unable to figure out how to undistort the raw color image using the camera intrinsics.
With a lens distort, and using invert theres a huge ring around the image center, with glsl its so warped edges move into the center, and the pixels in the center move to the edges, and with just a raw opencv undistort function i still cant seem to get the undistort properly.
Sorry for the delayed response. I don’t really have a quick answer for you, so it requires finding some time to dig into it myself which is always scarce. I’ve got a little time tonight and I will take a look again and see if I can help move things along for you.
In theory you could use the Kinect Azure class function transformWorldToColorImage to take the projected depth point and go directly to the UVs of the color map. It might not be the fastest to call it for every pixel, but it might be a decent test of the theory.
I’ll play with it some more tomorrow and let you know what I find, but hopefully this helps a little.
First, thank you! I appreciate the support.
If this function does everything magically, that’s even more amazing, and if its slow that’s fine, I’m happy to have a script to do it in a single passthrough in TD for every frame of a movie file.
But I cant even seem to get a positive return from the worldToColorImage(tdu.Vector) function
Sorry to pester about this thread! Are there any examples for how worldToColorImage() can be used? Whenever I plug in a 3d vector with xyz coordinates like from the point cloud directly or even just arbitrary numbers it only returns false for me.
Is it just a function from the azure TOP? or is there some sort of reference between the input vector and the current image displayed in the TOP? I would like to take every pixel from the depth point image and place them onto a canvas at the resolution of the color image if possible!
