Run out of gpu memory error crash

POPs
,
1

Generally speaking:
I have a blob tracking TOP system in a COMP that creates tubes which point from the blobs coords to some other coordinate depending on parameters. This comp is controlled by an extension which handles various state changes. I used most of the comp for a similar project where my tubes were created in TOPs, and am simply attempting to update to a pops workflow. Never had any issues with older version.

I suspect the issue is with the pops network, which is outlined below.

  • The target positions are calculated in DATs stemming from the blobtrack info dat.
  • Then using a dat to pop to create attributes and direction
  • three glsl_copy pops then runs a simple, but different bezier functions on a line pop, which glsl pop is determined by a switch pop
  • I then copy circles at each point in the lines, and skin them creating tubes

Running on a 3080 laptop, Windows 11, latest TD build, I am consistently crashing with the error.

“Touchdesigner has run out of GPU memory and must close. Try to reduce GPU memory usage in your project to avoid this error.”

But the gpu memory icon in the toolbar states that I’m only using 0.5-0.6gb of gpu memory.

Crashes happen on state changes, like when opening my python extension, or triggering a state change etc. Once a crash has occurred the file will not open and only gives the gpu memory warning before failing.

I can create my own CrashAutoSave file, disable the pops network, resave and get it to reopen, but is still very unstable.

TD doesn’t create a Crash file or the .dmp file

2

Hello @electro666

Thanks for the detailed report. I’ve been testing with the file you shared, including triggering state changes and re‑initializing the BlobsExt extension, but so far I haven’t been able to reproduce the crash. If you have a specific sequence of actions that consistently triggers it, that would help us narrow things down.

One way to check whether the GPU is actually running out of memory is to use the Windows Task Manager’s performance view. If you select your NVIDIA GPU there, you can monitor VRAM usage over time and see if any spikes occur around the moment of the crash.

1 Like
3

Thanks for the reply Guillaume.

Yes I can recreate.

If you go into
/project1/blobs_ultimate_r/blobs_format_change/dat_format

and remove the select DAT named s_limit

And then run the state_1 DAT it will crash.

Additionally, I am in the middle of trying these things to solve the issue:
I noticed the extension runs a state on _init_ ,which I have since updated/moved to onInitTD()

As well as using a selectDAT to limit points feeding the POP system, although im unsure why the crash is occurring.

I have also merged the three glsl_copy pops into one operator as to bypass the switch pop.

4

Spent a bunch of time looking into this more and was able to further isolate some things. Although, still not sure if it’s a bug or misuse. I’m no DAT-ologist :wink:

  • Between my extension and parexecute there was a small recursion, so I rewrote and solved that problem. But upon finishing that there was still instability btwn specifically with the blobtrack TOP.
    At first I thought this was because I wasn’t limiting the amount of blobs coords that get turned into POPs, but when I limited it to small number 1, 4 etc. Still immediately crashes on state change and on blobtrack.par.reset or reset.pulse().

  • Replaced the blobtrack → DAT → POP pipeline with blobtrack → CHOP → POP. Seems more stable, but still testing.

I’m including two files:

  1. StableCHOP2POP.toe : blobtrack → CHOP → POP (no DAT)
  2. UnstableDAT2POP.toe : blobtrack → DAT → POP
    2a) Recreate by touching any reset par on the blobtrack.
    2b) State changes cause crash, but not consistently.
5

Thanks for sending the new samples. I was able to reproduce the crash immediately by opening BlobsUltimatePOPs.UnstableDAT2POP.toe. The crash is coming from a DAT-to-POP node (/project1/blobs_ultimate_r/bezier_tubes_pops/datto1). We have a fix lined up for the next build.
In the meantime, a workable workaround is to ensure that the DAT-to‑POP node always receives at least one header row. I updated your example to demonstrate this approach (see blobs_ultimate_r/blobs_format_change). Please let me know if the crash still occurs with this setup.

BlobsUltimatePOPs.UnstableDAT2POP.Modified.toe (44.6 KB)

1 Like
6

Thank so much Guillaume,
Making sure the header stays in the DAT does indeed solve the stability issues.

1 Like