What does the adoption of Vulcan mean for the future of TouchDesigner?

[lucasm]

Greetings Derivative!

I keep hearing mentions of the upcoming adoption of Vulcan as a big improvement , or at least change to the OpenGL side of things. However I still can’t find much official info on what this actually means for TouchDesigner.

Is there any info on what features or changes we can expect from this?
Also, is there a general timeline on when this might be expected? Sometime in 2020? or beyond 2020?

I almost don’t know what more specific questions to even ask, as Vulcan seems so far removed from the higher level aspects of Touch.

What I read on the wiki is that Vulcan is a really low level graphics API that supposedly makes use of cpu/gpu more efficient and balanced.

Are we just going to get a souped up TouchDesigner in TOPs and GLSL? Or are we also getting new features sets of some kind? rendering enhancements, etc?

Would love to learn more about what this all means for us
Thanks!

[malcolm]

Good question. It means a lot of things.

The first most basic reason to move to Vulkan is to greatly reduce GPU driver overhead. One of the biggest costs on the CPU side when running a complex .toe file is time spent in the graphics driver telling it what you want the GPU to do. This is time spent on the CPU, and is visible in the CPU cook time of TOPs. This is very expensive in OpenGL due to how complex and bloated the API has become. The driver does a LOT of work for you. With Vulkan this work is put onto me instead, and I can streamline it for our particular usage pattern. TOP CPU cook times are often about 50% less on the Vulkan version than on the OpenGL version.
Note that this does not mean that the GPU costs have reduced. A shader takes the same amount of time to execute on OpenGL as it does on Vulkan.

Next, feature parity. It helps in a few ways. On the Windows, I’m hoping things such as Texture Instancing will work on AMD cards. GL features that don’t quite cut it in the vendor-agostic extenions are designed better for Vulkan.
Targeting Vulkan allows us to use Metal directly on the macOS side, so now we’ll have faster and more modern graphics support on macOS as well. Their OpenGL support is getting worse it seems. This also sadly means we do lose some features, since Metal on macOS straight up doesn’t support Geometry Shaders at all, so those are going away on the macOS side.

Moving to Vulkan also frees us up to start designing new rendering systems. A lot of work wasn’t put into rendering lately due to not wanting to have to re-do things for Vulkan. Once we are onto Vulkan, which I expect to be the API for the next 20 years, we are free to really dive into it.

It also seems like Vulkan is branching out beyond graphics and adding support for more multi-vendor features such as Vulkan Video which will allow us to use GPU decoding/encoding of H264/H265 with a single API vs having to do it separately for Intel/AMD/Nvidia.

Happy to answer any questions you have!

[lucasm]

@malcolm thanks for the deep answer here, this is super clarifying already.

1. CPU side performance: So for example with a render top, the cpu cook time of loading all the geometry, shaders, and other things that happen every frame, that grows considerably when the number of objects reach higher and higher numbers is something we can expect to perform a lot better with Vulcan?

2. Windows/Mac compatibility: So essentially maintaining and building for both OS’s get’s easier and more streamlined it sounds like. Can see the value there. I can relate to the texture instancing woe, I’ve for years now straight up avoided that technique 100% because of how many folks have emailed me in the beginning saying icons and graphics in the UI aren’t showing up.

3. New rendering systems you say?: This part is what I personally am most excited about hearing more on… I know it’s early and things may change a lot internally, but do you have any general / vague roadmap of what you mean by new rendering systems? And also, is there a rough timeline on what to expect? Trying to gauge how much to build on my own in the mean time, or if there are some exciting new things to expect soon ish?

Thanks for all the clarity! Understand if some of these Q’s can’t be answered entirely.

[malcolm]

Yep, more complex scenes with lots of different parts should have lower CPU cost associated with them now.

Stuff like deferred rendering, using the new raytracing APIs. I’m not sure what we’ll do yet but there’s no reason not to push ahead once Vulkan is done.

Vulkan is very very far along btw. I’ve been heads down on it for a few months now.

[lucasm]

Awesome, well the cpu bottle of render tops I am super excited about. that comes up a lot as an optimizing vs elegant object oriented scene setup consideration. I batch things together when possible, but still can get heavy with lots of elements that can’t be easily batched together.

Looking forward to that improvement when it’s ready Malcom, thanks for the information.

[raganmd]

@malcolm does the transition to Vulkan open up any space for re-thinking the architecture of SOPs? I could imagine a future where those might be partially driven by compute shaders under the hood, and I’m curious if there’s room for those kinds of feature requests in the not too distant future.

I think I’ve read that there’s some advantage to better instruction distribution across cores (at least on multi-core CPUs) with the Vulkan API. I’m also curious if this means we might see some performance benefits with respect to hardware that has many slower cores vs fewer faster ones.

This is one of the most exciting projects at Derivative that you all have been working on lately, and I have to admit that I’m very excited to see how this all evolves.

[malcolm]

Yes, definitely. That’s another feature parity it gives us with macOS. We’ll finally have compute shaders on macOS so we can start using them heavily.

[nettoyeur]

Thanks for explaining all that Malcolm. Do you think using Vulkan could offer new possible options for developing any multi-GPU capabilities for TD? For instance fast texture sharing between multiple gpu’s, is that easier in Vulkan?

[malcolm]

Multi-GPU support is definitely way stronger, in that addressing many GPUs has a saner API now.
I don’t think copying between GPUs is much different as the reality of the hardware transfers speed is still the main issue there.

[elburz]

This is awesome to hear. Congrats on getting far into the Vulkan version. I was imagining it might take a lot longer but sounds like lots of progress being made.

I haven’t followed Vulkan too intensely, but I’ve heard some of the benefits. One thing that peaked my interest are the translation layers, which sounds similar to what @malcolm was saying about feature parity being better between Windows/macOS. Does that mean there are some possibilities to more easily transition to other platforms in the future?

Linux being a big one folks are always asking about, but I’m even thinking about the possibility of having TouchPlayer mobile (if there are translation layers for that kind of thing) that can run really simple projects with a UI and some OSC out or similar (even heavily feature-scoped down version of TouchDesigner with a few UI comps and a few CHOPs/TOPs would be pretty amazing I think). Similar question would be about web compatibility layers. I haven’t followed webgl or webgpu much, but I’d be equally curious if similar stripped-down TouchDesigner of only a few UI comps, basic CHOPs, and some basic TOPs, could be made to playback small interfaces in the web.

[malcolm]

It doesn’t help with Linux porting. Linux always had great OpenGL support so the OpenGL renderer would have ported without issue, and the Vulkan one will port without issue also.

The big cost of porting to platforms is the windowing system and I/O (mouse, sound etc.). Also the extra cost of dealing with maintaining all of those platform specific issues.

In general the graphics API isn’t a hurdle when porting between platforms for the most part (unless you are using DirectX)

[art3mis]

Hi Malcolm. Other than improved performance and feature parity are there any other specific features we can look forward to?
Is it possible to give a hint as to when this might be released?

[malcolm]

Nothing more specific yet. The first release is focusing on just getting everything working and up to speed. So the first release won’t be splashy. Aiming for the 2020.60000 series of builds though.

[vinz99]

Thanks for all the info!
Regarding the loss of geometry shaders, just wanted to point to this funny tweet

()

Would love access to geometry creation with compute/compute buffers though

(edit just realized the above tweet is more relevant for smartphones GPU, I did remember reading a few times geometry shaders performance could vary greatly on desktop as well)

[mynameiscorey]

Hi Malcom, thanks for all the detailed responses so far.

I was just wondering, since you are controlling more of the work that was previously left to the driver, could there be an opportunity there to expose some control over flushing / preloading GPU memory for TOPs?

I’ve had problems with projects that use a bunch COMPs for scenes of generative content, each containing several TOPs and sometimes render networks. The first time you point a selectTOP at the output of one of these COMPs, Touch of course has to cook everything inside and allocate GPU memory as needed, but in most cases this happens without frame drops.

The problem I often face is that by the time you select maybe the 50th scene COMP, you’ve run out of GPU memory and Touch hangs for several frames while (I’m guessing) the driver decides what to purge to make room for your new buffers.

The only technique I’ve seen to counter this issue is to change the resolution of TOPs you are finished with to 1x1 and then resize them the frame before you need them again. Could it be possible after Vulkan to better manange GPU memory with Python with something like an .unload() method in the TOP class or an .unloadChildTops() method in the COMP class?

And I might be venturing into dreamland now but how about a .preload() method for TOPs that allocates GPU memory and does all the CPU work that is common to all TOPs, but stops short of a full cook and execution of the TOP’s specific shader functions? And maybe a .preloaded attribute you could check before pointing your selectTOP at a TOP?

Thanks for the great updates Malcom. Hope I haven’t derailed the thread with my uneducated guesses and ideas!

[malcolm]

That kind of stuff isn’t really Vulkan related and more just general programming. Adding preload() to all TOPs is possible but just requires a large rework of how TOPs cook. The memory allocation tends to be the expensive operation though so avoiding the shader process when preloading isn’t going to save you that much.

[mynameiscorey]

Sure, fair enough. Thanks for the explanation.

Is there a doubling down then in the hit to performance when GPU memory is already exhausted and you ask Touch to allocate new memory for some TOPs? This seems to be where things fall over for me, not surprisingly.

Is memory intelligently purged based on what is being cooked / displayed or it is it a simple FIFO situation? Is Touch handling this or the driver?

Cheers.

[malcolm]

With OpenGL the driver handles it. With Vulkan I need to. It’s likely that kind of situation will be worse in Vulkan, at least to start, as there is a lot of logic/tech behind how the driver writers for OpenGL handle it that I’ll need to recreate from scratch.

[lucasm]

On the subject of deferred rendering, what does this mean for the future of lighting / shading?

Specifically, I’m wondering about the 32 light comp limit per render TOP - will this still be a hard limit still for deferred? Or will it be possible to surpass this limit which will in turn make it easier to build more advanced lighting optimization systems and effects on top of the current system?

[malcolm]

That limit goes away with deferred rendering since you accumulate lights using multiple passes internally (if you don’t have enough resources to calculate them all in one).