I’m now searching for a way to visualize and manipulate them in realtime inside TD.
The training process requires a lot of libraries and dependencies, but at the end, we just obtain a *.json file (for the cameras) and a *.ply file (with a lot of points attributes). Indeed, the ellipsoids have to change shape, color and opacity depending of the camera position.
Quite difficult, BUT… Inria also provides the sourcecode and windows binaries for a viewer, based on their SIBR framework. And it’s really fast !
I am also very interesting in getting Gaussian Splats to work in TD.
There seems to be a Unity addon already created and previously I experimented with the NeRF Studio add-on for Blender. It is only a matter of time until someone takes up the challenge to implement this in TD. I guess the key to getting this to work inside TD would be to understand how this SIBR viewer works and translate it into python code or a custom operator?
Not sure, but hopefully more knowledgeable people can people start chiming in to get the ball rolling with ideas.
I have to tell, with @timgerritsen, we have made some major steps to bring 3D Gaussian Splats in TD. (I say we, but in fact… he did ^^)
Tim used this output file as well and he wrote a shader to handle the positioning, the coloring (difficult task with color harmonics) and the display of the gaussian splats.
The points in the ply files contains a LOOOT of information, we extract with PointFileIn and multiple PointFileSelect
Not perfect yet, still some size and orientation issue, but very promising results with a very good performance !!! See the image below, rendered in TD, with a 180° fisheye from cubemap.
Yes we’ve been diving deep in the splatting business Very interesting technique indeed.
As far as we understand now is that the PLY file contains positions of the splats (xyz), rotational information (quaternion), scales (in log space) and spherical harmonics to describe the colors depending on the viewing direction. We’ve managed to load all the data though still bumping into some problems. It feels the splats are not orientated correctly if I compare it to the SIBR viewer. Might be related that the coordinate system is different to TD, but haven’t figured out yet to get the right orientations.
Also a similar problem we have with the spherical harmonics, giving weird red/green-ish colors when using higher orders of the harmonics (the fish eye image only uses the first order, so kinda like a diffuse color for all the directions).
Anyways will dive into it some more and keep you posted
Fun times! I’ve got my system setup to produce these also. Was looking into the unreal plugin but price is steep. Let me know if you’d like anymore test files.
Quine
Very interesting technique indeed.