basically i have a media system that is loading and unlaoding files into Mediafilein TOPs, after my system is running for a “long time” and after multiple switches of moviefilein files.
i get performance issues with the moviefilein top.
fresh load, i get about 0.2ms on the cook, after i have switched files, reloaded, maybe some time has passed, the movie has looped a lot…just normalish use, the tops jump to 4ms cooking.
restarting the project revolves it.
i added a script to .unoad() when i load a new file into the top. but it doesnt seem to help.
is there another way to clear chache, or reset the nodes cook, or i dunno…each time i change the file in and reload? to ensure im not getting hits over time/when loading lots of different media..
Im noticing when it happens, that my cpu and gpu are not in the highs of utlizition.. but i go from like 15ms to 60ms for no apparent reason besides for havign switched out and reloaded clips on the moviefilein tops.
when running .unload(), are you using the cacheMemory argument? It can be useful if a different Movie File In TOP will be later opened with the same resolution/codec.
Otherwise, can you see if GPU Memory is rising over time and reaching your video card’s limit?
Any chance to see what is causing the high cook times? Is it when opening the file or when reading frames?
What TouchDesigner version are you experiencing this with and how many videos are playing at what resolution on what hardware?
I haven’t put the argument in but will.
So would it be:
unload(cacheMemory=True)
?
As for GPU memory it’s not hitting its limit, neither is CPU or GPU prefromance, that’s what’s odd about the whole thing.
I’m on the most recent build, I have dual a6000, where the project that is causing the issue has its affinity set correctly, I’m playing 9 4k nothc lc videos, frame rates varry but mostly at 23.98.
As mentioned they play fine and smooth when loaded, but if I switch out the files with new files, or time passes…there’s no real rep repo here unfortunately, but at some point prefromance just gets hit and we go from a smooth 12ms to 60+ms. The biggest spike I can see is with the moviefilein tops where they are now cooking much more then they were/should be.
It’s not when they load it’s when theyre playing
Do you have a test.toe where I could try to recreate the problem? If you use NotchLC I guess you have the highperformanceread option enabled, right? Depending on your SSD and cooling maybe an overheating drive might be an issue?
Maybe you could try to wrap the moviefileinTOP in a comp, save it as external tox and reload the whole tox, just to see if it helps with the performance issues.
i’d remove unload() / unload(cacheMemory) completely if you’re using the same resolution/codec in your moviefileinTOPs as preload() will re-use the memory automatically. running unload() forces a re-allocation every time, which is less than ideal.
how’s your operator playback system / control setup? you mentioned you’re playing 9 videos across multiple instances? can you explain the full setup in more detail? are you running a mosaic, syncing data between instances, etc.? are you using a bunch of moviefileinTOPs? i’d recommend a basic A/B deck system with just two moviefileinTOPs and just swap out the files realtime and run preload()
also, have you tried using h265 w/hardware decoding on? with your a6000s it should be quite performant. i basically use h265 exclusively now with similar multi-a6000 GPU setups with 8-16k resolutions w/out issues. it usually just comes down to finding the right balance of h265 encoding settings for the hardware setup - re: bitrate, etc.
lastly - have you looked into using the Direct Display Out TOP? if you have Windows 11 Pro for Workstation you can bypass Windows altogether and output directly to the GPU - ie. no perform window. i’ve been using this lately and it’s a game changer in terms of performance…
can you provide any sample .toe / code for review?
Hey,
So it’s one instance, and 9movieinTOPs,
We don’t want to use h264/5, it’s not a good standard for the color space and clarity we need. It’s a very last resort. Also notchLC is way more preformant afaik.
We have a very robust library system, that replaces the file paramter of the moviefileInTOP. And the unload does seem to help, it’s fine that it’s more processing when loading/unloading as the “transition” isn’t important, I just need smooth playback once files are loaded.
It’s a 8k mosaic, one one GPU, synced with a Quadro card.
But we are loading/playing all 9clips at once, but have never had any performance issues, and as mentioned it’s not even hitting 50% vram or utilization.
As for direct display out, - I wish…but we don’t have pro, I may upgrade for this as that would be nice to have, but I don’t really see it as a output problem, because it’s happening when switching the file paramter, and oddly enough, some files play ok and some don’t.
That’s why I just want a very robust clearing of any cache or resources evrye time I change the files/load a new folder.
copy - ya notchLC will be technically more performant but at the cost of vastly larger file size. h264 is unviable - i’m talking exclusively about HEVC which actually does have solid color space flexibility. understandable if it’s not viable in your case though.
is it possible in your setup to pack multiple videos into a single video - ie. (6) 4Ks into (1) 7680x6480? you mentioned the transitions aren’t important - so potentially you could bake the crossfades for each video into the packed video(s) and reduce your moviefileinTOPs down significantly from 9. i suspect having that many moviefileinTOPs could be causing an issue.
also, you mentioned performance has been fine - however, you could split your system into (2) 4K mosaics and run (2) TD instances bound to each GPU via GPU affinity for absolute maximum performance overhead. not sure of your physical display setup, but if you require sync between instances and no Pro license for Sync CHOPs, since it’s just movie playback, you could drive a shared movie index between instances via SharedMemCHOPs set to Global. i’ve done that in the past and it’s worked nicely.
if this setup is feasible w/regards to your content playback requirements, etc, - each TD instance could be playing back up to (6) 4K videos using 1 moviefileinTOP per instance, essentially splitting the workload up as evenly as possible across your system.
I will take a look at hvec, you haven’t had any issues with color, banding, compression. We are running a 12bit 444 output and do color correction live, so need all that information.
Would be very happy to move away from notchLC, files sizes are ridiculous and also I personally don’t do them that preformant, just better then the alternative “uncompressed” codecs.
I can’t/don’t want to pack files, we deal with lots of driving plates and footage, adding a layer of encoding in addition is a lot, in theory I could create a template but I just don’t see why playing back 2 moviefileins vs 9 if it’s the same resolution would make a difference.
Would be happy to hear about that from someone on the derivative crew though…
I have flirted with having two instances of touch running and then using sync chops. But we very rarely have had issues. And the added complexity of that doesn’t make much sense when our cpu/GPU isn’t hitting anywhere near its potential, it does feel like either soemthing wrong with my logic/setup/system design or a wired bug that is tied to either a driver or the way TD is handling playing back footage.
The biggest indication to me is that a. The system isn’t at max at all and b. It’s solved by restarting touch…so it’s almost 100% software related - either with my design or with TD…
I’m wondering if having multiple touchIn and Out TOPs could be causing issues as that’s really the only major new change in the system that I can think of, besides that we have been stable and preformant for over a year.
I have some performance logs, one from when it’s chugging and one from when it’s smooth. If anyone wants to take a look. I’ll upload them in the next post
i do have a chop that is processing the length and frame index of the moviefileinTOPs and it gets its length from the cam01 moviefileinTOP.
maybe i need to clear this chop or somehow make sure its lengtrh is not being effeced by loading and unlaoding files?
but it also seems to be due to the local/time/clock - why would this be altered over time. and how can i make sure its sample count/length doesnt change over time?
no quality issues if you have control of your source media and encoding quality. ProRes444 source content and FFmpeg for re-encoding will give you plenty of options for quality/performance and flexibility encoding to h265 12bit 444.
i typically use the ‘power of 0.75’ rule as a general guideline to determine optimal/performant bitrate for resolution → New Bitrate = Base Bitrate × (Resolution Ratio)^0.75
i use a base rate of 9-27 mbps 1080p 60fps. in your case, extrapolating to 4K with ~3x buffer for the added 12-bit 4:4:4 requirements = ~75-228 mbps. this would be a good starting point - you can adjust preset, bitrate, constant vs variable bitrate/crf settings to find what works best for your setup - ie:
without some more context via a .toe or any code to review - i can’t speak much to the logs. however, i don’t think local/time/clock is the root of the issue. you mentioned using multiple touchin/out tops in the network - what’s the point of those if you’re running only a single instance? also, i see in the log analysis you’re using a video device out top as well? seems like there is a lot more going on in the network than you’ve described.
if you could provide more information or the .toe it could help clear things up
Reloading the tox might be like restarting the project for the moviefileinTOP, but not for everything else. In an ideal world this might already fix your problem and point towards the moviefileinTOP. If it doesnt solve the problem it might be somewhere else (maybe your logik, maybe “the rest of touchdesigner” - who knows).
We are using multiple servers with several 3860p60 video encoded as HAPQ and/or h.265 and its been running fine for many months now with only restarting the servers every weekend. However, as we build a content scheduling system, we are unloading the movies and setting the moviefileinTOP to bypass a lot and thats where my idea came from.
I just had a look at the logfiles you´ve posted, but I´m not sure how to interpret them? The Working2.txt contains moviefileinTOPs with very different namings and paths - are those from the same patch?
Do the framedrops also occur in perfom mode? And could you state which operators are reading from and writing to the moviefileins?
… and just to be sure we are on the same page regarding the directdisplayoutTOP: You dont need a TD Pro license in order to use it, but Win 11 Pro for Workstations. I even came across the limitiations of not beeing able to use “spare” outputs of a GPU when using a mosaic: As soon as you remove a mosaic from the desktop the remaining ports are getting reactivated.
However, if restarting touchdesigner immediately solves the problem and you are running into the problem consistently (and for more than one frame?) it might indeed point somewhere else. I havnt really worked with touchin & outs but that might be worth a look. What exactly are they controlling?
Hey, I don’t think AI tools are useful to analyze things like this. The output it’s giving isn’t really correct at all. The values the performance monitor aren’t the number of samples in the CHOP, it’s simply the cook count. So that number will grow endlessly, and expect to be high when the file has been running for a long time.
If there aren’t any nodes taking up a a significant amount of time, but things are just overall just acting slower, I would wonder if there is a thermal throttling occurring on the hardware after it’s been running for a while.
Does things slow down after a consistent length of time? How long does it take to before performance degrades?
hey,
thanks for that ffmpeg scritps, will def test this in the future!
i am running content from a single instance and a control/ui patch on a seprate instnace, they are sharing dats, some chops and some monitoring tops.
video device out is just sending a confidence monitor out.
this part of the project is playing back multiple plates and reprojecting them onto geomtry and then outputing them. it all goes through a media server and color correction pipeline.
putting this part of the chain into an external tox is a very intresting idea and wouldnt be so difficult, and i can just reload the tox evrey time i hange the file paramters.
i just dont understand why the derivite crew isnt piping up about a way to do this WITHOUT that workaround..
i mean shouldnt every node have a “reset me/unload/grabage dump” funciton, and they do rihgt like the unload(), so why doesnt that work but restarting touch does..i dunno its just weird.
i have had this setup running without issues for over a year, playing b ack 9 4k notchLC videos with no problem… i added some new logic, that seemingly works and now i get these periodic issues, but i cant seem to see where in my new logic the issue would arrise and most of the new logic is on the control instance not on the actual probelmatic instance…so its frustrting.
its all in prefromance mode.
yes i understand the displayoutTOP, we dont have pro windows, first of all we are on 10, second of all its a super specfic branch of 10 (the pro workstation) and its just not what we have, i could upgrade them i geuss, and we may, but i dont see how the outputing mechanisim would be cuasing the issue to be honest.
also i would still have to use mosaic with directdisplay as the limitation is currently only 3 montiors and i need to output 4, so i would have to use mosaic as “1 monitor” - mosaic is fine for our setup, actually makes things a bit simpler and easier with the absloutlty retarted way windows handles display postions…which yeah directdisplay would solve as well haha.
touch in and out, is just sharing some montioring textrues and previews, they are pretty low resoluton, nothing over 1920.
i was scrambling to find a solution and get logs during a production and was not getting any viable help from support…so…figured id try my luck with our ai overlords…
but yes i see your point and realize now its kind of misleading, it did help pinpoint where the ms are ramping up but i already knew it was all hapepning in the moviefieTOPs.
so yeah…
there is no thermal throttling happening, i can confirm this, and as mentioned, when i go from 12ms to 60+ms cooking time, my systme, both gpu and cpu is at 40%…and my harddrvies are not changign either.
its not consistnet. its seemingly random but does seem to occur when we do a lot of back and fourth library switching, so loading and unloading files on the moviefileinTOP.
Unless you are running out of memory, which currently does not seem to be the case, cleaning up nodes won’t change performance. Even if the nodes have lots of garbage/unneeded data, the existence doesn’t affect the performance, until memory is becoming constrained.
Ya in this case I don’t think that’s the issue, so it’s premature to try that out.
I’ve been following the thread. The problem is it’s still unclear what is causing this degradation still. I continue to read through your posts to see if anything comes up.
