3D Gaussian Splatting?

Hello. I’ve started experiment with 3D GAUSSIAN SPLATTING

I’ve followed this method, and I’m totally hyped : GitHub - graphdeco-inria/gaussian-splatting: Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering"

I’m now searching for a way to visualize and manipulate them in realtime inside TD.

The training process requires a lot of libraries and dependencies, but at the end, we just obtain a *.json file (for the cameras) and a *.ply file (with a lot of points attributes). Indeed, the ellipsoids have to change shape, color and opacity depending of the camera position.

Quite difficult, BUT… Inria also provides the sourcecode and windows binaries for a viewer, based on their SIBR framework. And it’s really fast !

Here : https://sibr.gitlabpages.inria.fr

Do you think I can run it through TD?

Or maybe recreate this method via a shader?

Any ideas? Thanks

5 Likes

I am also very interesting in getting Gaussian Splats to work in TD.

There seems to be a Unity addon already created and previously I experimented with the NeRF Studio add-on for Blender. It is only a matter of time until someone takes up the challenge to implement this in TD. I guess the key to getting this to work inside TD would be to understand how this SIBR viewer works and translate it into python code or a custom operator?

Not sure, but hopefully more knowledgeable people can people start chiming in to get the ball rolling with ideas.

@b2bk,

would be interesting to see a resulting .ply and json file - do you have one of those?

cheers
Markus

1 Like

Hi @snaut , hi @julsbell

Markus, I’ve uploaded the output file i used for this video (a metal workshop near Amsterdam)
The archive link (expires in about 30 days) :
https://www.swisstransfer.com/d/64847d69-18d9-4a5b-bf75-d073beb3990d

The preview video :

Screenshot taken in SIBR viewer :

I have to tell, with @timgerritsen, we have made some major steps to bring 3D Gaussian Splats in TD. (I say we, but in fact… he did ^^)
Tim used this output file as well and he wrote a shader to handle the positioning, the coloring (difficult task with color harmonics) and the display of the gaussian splats.
The points in the ply files contains a LOOOT of information, we extract with PointFileIn and multiple PointFileSelect

Not perfect yet, still some size and orientation issue, but very promising results with a very good performance !!! See the image below, rendered in TD, with a 180° fisheye from cubemap.

I’ll keep you posted, but i can say already we really love this technique, and will try to go as further as possible :slight_smile:

8 Likes

Hi there!

Yes we’ve been diving deep in the splatting business :smiley: Very interesting technique indeed.
As far as we understand now is that the PLY file contains positions of the splats (xyz), rotational information (quaternion), scales (in log space) and spherical harmonics to describe the colors depending on the viewing direction. We’ve managed to load all the data though still bumping into some problems. It feels the splats are not orientated correctly if I compare it to the SIBR viewer. Might be related that the coordinate system is different to TD, but haven’t figured out yet to get the right orientations.

Also a similar problem we have with the spherical harmonics, giving weird red/green-ish colors when using higher orders of the harmonics (the fish eye image only uses the first order, so kinda like a diffuse color for all the directions).

Anyways will dive into it some more and keep you posted :smiley:

cheers,
Tim

8 Likes

Fun times! I’ve got my system setup to produce these also. Was looking into the unreal plugin but price is steep. Let me know if you’d like anymore test files.
Quine

Hey Markus,
I’ve got a bunch of files also if you need anymore.
Quine

Hi man! This is sooooo dope! I’ve started doing some experiments to but I can’t figure out how to set the colors properly. May I ask you if you have any tips that can help me? That would be great

Few days ago a paper and github repo on 4D Gaussian Splatting was published.
So now with time as the extra dimension - which means capturing motion and animating it! 70 FPS at a 800*800 resolution on an RTX 3090.

5 Likes

I’ve had some success through instancing from the PLY file. It seems that there are colour values in the .PLY (I forget the name but there are lots of them) and there would need to be a way to merge them all together to properly work. Any 3 will colour the instances, but only partly and it reacts a bit weird.

I’ve given up and I’m using a Blender plugin (https://www.youtube.com/watch?v=yKz7OfomyCo)

It has an option to export as a new .PLY which I am yet to test but this might merge all the colour data to work nicer with Touch.

Those color values are blended by spherical harmonics. So every splat can have completely different colors depending on the viewing direction.

See for example https://www.researchgate.net/profile/Jacob-Hollebon-2/publication/345372557/figure/fig1/AS:954811419725829@1604656190963/Real-part-of-a-set-of-spherical-harmonics-mapped-to-the-surface-of-a-sphere-The-colour.ppm

I think there are 16 harmonics but you could just take the first one (f_dc_[0-2]) as a color which is kind of the average of all directions.

Cheers,
Tim

Hi Tim! Congrats for your work.
I’ve been diggin a bit into this as well and, as you said, I get some strange orientation.
Did you manage to understand what’s wrong?
Also, I was trying to sort on the GPU using bitonic sort (kind of started from here. How did you manage that?
Thanks for sharing!
Best.

Hey Matteo,

I found the camera orientation, but unfortunately at that moment I couldn’t get the gaussian splats orientated and scaled as other viewers. I haven’t been trying since.
If you rotate the camera over the x axis 180 degrees, everything should be at least on the right position :)… but I guess somewhere in my covariance calculation I’m doing something wrong.

Yeah I was also using a bitonic sort, which becomes quite heavy when you have a lot of points. So I think we would need to implement some filtering to not sort all the points every frame. But I decided to think about that after solving the orientation problem and just used a raw bitonic sort over all points.

I think lots of opensource viewers popped up in the meantime, so perhaps we can find the solution there :slight_smile:

Cheers,
tim

ps. I’m also on the touchdesigner discord if you want to discuss it in more depth

1 Like

Hi there! Really struggling with these Gaussian Splats…
I found this usefull tool: playcanvas.c o m/super-splat
to edit, clean and scale the splats.
Finding a way to add colours to the splat in TD but nothing…

Have any of you found a way and can share a short walkthrough or sample project? :cry:
I found the work of @yeataro on X that is really impressive…I read that he was planning to release the project soon but nothing more has been seen.

Please master yeataro help us <3
Cheers
Andrea

hi Tim,
sorry for the late reply, busy days :slight_smile:

I spent quite a few time on the code already and couldn’t figure the mistakes out. Maybe we just both messed some matrix multiplications, but my best guess so far is that there’s something going on with the differences between Vulkan and other graphical APIs. For example, clip space is similiar to DirectX (0-1) on the z axis, and the y axis is negative. So maybe somehow this should be taken into account in the vertex shader.
A few resources I had a look to - need for sure to dig this a little further - this one and this other one. But well maybe that could be it, I’m really not sure about this one…
What do you think?

Hi there! Did anyone had luck with this?

I would love to play with gaussian splatting files in TD.

Hi there,

Yes I finally got lucky and found what was wrong in my code. I’m planning to clean up the stuff this week and will share it on the community page.

cheers,
tim

4 Likes

That’s great!
Any chance to get a spoiler about what was wrong? Maybe I can figure it out by myself as well :cry:
thanks

For me it was that I didn’t normalize the quaternion.
I just shared my version on the community page.

4 Likes

Thanks Tim,

This week I managed to understand how pointclouds and color selection worked and managed to see the point colors, rotation and scale.

But the component you have made looks fantastic.

Thanks for sharing!

1 Like