How to store large data in TD

Hello,
please may I ask what would be a recommended approach for dealing with large data inside of TD? I would like to store xyz and rgb data for pointcloud that is being constantly updated (new and new points coming in). Using DATs works fine but after a certain point it is too heavy to add new rows with data. I guess there are much better ways of dealing with large amount of data so I thought I might ask here for help. Thank you very much :slightly_smiling_face:

You may look at the point file in TOP - you can author .ply and .xyz files with python. Depending on the size of your data set and the frequency of updating this might be a way for you to work.

Thanks, but unfortunately this is not an option for me as new points are coming in on each frame. Therefore I am looking for a way to store these data directly in memory. :slight_smile:

In the latest stable I think there’s now a script TOP that handles a TOP as a numpy array. I don’t know what the speed of that might look like (I haven’t done any testing yet) - but that combined with a feedback TOP might let you think of using the script TOP as a way to add a point to an existing buffer? That way you’re only ever writing a handful of pixels (which should be cheap), and then caching them into a TOP pipeline.

That sounds pretty cool. Thank you very much. I will test it see how fast it would be.

Or bypassing python, your data has to come in live somehow, say in a DAT as a table…, or in a CHOP…

If it’s arriving in a DAT, you can get it into a long set of CHOP channels using the DAT to CHOP.

Once it is in CHOPs you can get it into TOPs using the CHOP to TOP, and you can use the new feature of that operator where you can pack all your XHOP samples into a square image, it figures out what resolution automatically, and then you can use TOPs to manipulate the data as pixel-per-xyzpoint.

Thanks, that sounds great. However I should have mentioned that these points are coming in through UDP In DAT so I guess it won’t be possible to bypass python as it is handling callbacks…