Export Raw Pixel Data from Video Stream to File

Hi, I’m working on a project where I need to export raw pixel data (8-bit RGB color values) out of touch designer to later be read by a microcontroller using an SD card reader. Essentially I need a file that will just be a bunch of RBG values in sequence, with a header of consistent length. I’m not exporting very high resolution, probably 128x96 at the absolute most, and more commonly as low as 16x8.

I tried using the ‘Movie File Out’ TOP to export in the .bmp format, which gives me the type of data I need (can clearly see the different RGB values when I look at the file as hex), but only a single frame. I thought ‘image sequence’ might be what I needed, but this only exported a single frame as well.

Is there anything in TouchDesigner that will do this for me? To give a little background, I am very well versed in electrical engineering and writing firmware, as well as visual programming environments like Max, but am pretty new to TouchDesigner in particular. Any help is much appreciated.

hey epalaima,

you were on the right track, and almost there!

The Movie File Out TOP image sequence option is exactly what you need. All you need to do is to make sure the filename gets changed every time it writes a new frame - otherwise all frames get overwritten to the same file :wink: The easiest way to do this is to enable the “unique suffix” option. If you need more control about the file names you can change the N parameter yourself each frame.
See example attached - if you press start it will generate 60 *.bmp files on your disk, in the directory this .toe file is in.

image_sequence_demo.toe (5.1 KB)

The default expression that’s in the Movie File Out TOP does this all automatically for you
'TDMovieOut' + me.fileSuffix

Specifically, the me.fileSuffix will auto increment the suffix without having to do anything with N parameter or ‘Unique Suffix’ options. So as long as you maintain the me.fileSuffix suffix on your expression things will work.

Hmmm… I don’t know if that’s exactly what I need. I’m looking to create a single file with sequential image data, not a separate file for every frame.

I’m looking to record sequences that are several minutes long, and potentially longer ones up to 30 min. At 24 fps this would create a total of 1440 files per minute to record a sequence. Obviously this is pretty prohibitive and inefficient.

I get that putting it all in one file this will create a huge file size, but I’m using an SD card with GBs of storage so that isn’t a big concern.

I’ve seen some command line utility stuff that can convert .mov to raw rgb data, which is my backup if this doesn’t work inside TD, but it would be a lot cleaner if TD had a native option.

Ah sorry I didn’t get from your question you wanted a single file.
In that case I would record a movie and make sure you use a lossless and uncompressed codec such as Animation. Then you can read each videoframe from the file.

Animation codec may be uncompressed but it doesn’t appear to be raw rgb or rgba, which is more what I’m looking for for ease of debugging and programming.

An animation codec export looks something like this in hex:

> 0000 0014 6674 7970 7174 2020 0000 0200
> 7174 2020 0000 0008 7769 6465 0000 1436
> 6d64 6174 0000 003f 0000 01f0 ff40 80bf
> ff01 f0ff 4080 bfff 01f0 ff40 80bf ff01
> f0ff 4080 bfff 01f0 ff40 80bf ff01 f0ff
> 4080 bfff 01f0 ff40 80bf ff01 f0ff 4080
> bfff 0000 0000 3f00 0001 f0ff 4080 bfff
> 01f0 ff40 80bf ff01 f0ff 4080 bfff 01f0
> ff40 80bf ff01 f0ff 4080 bfff 01f0 ff40
> 80bf ff01 f0ff 4080 bfff 01f0 ff40 80bf
> ff00 0000 003f 0000 01f0 ff40 80bf ff01
> f0ff 4080 bfff 01f0 ff40 80bf ff01 f0ff
> 4080 bfff 01f0 ff40 80bf ff01 f0ff 4080
> bfff 01f0 ff40 80bf ff01 f0ff 4080 bfff
> 0000 0000 3f00 0001 f0ff 4080 bfff 01f0
> ff40 80bf ff01 f0ff 4080 bfff 01f0 ff40
> 80bf ff01 f0ff 4080 bfff 01f0 ff40 80bf
> ff01 f0ff 4080 bfff 01f0 ff40 80bf ff00
> 0000 003f 0000 01f0 ff40 80bf ff01 f0ff
> 4080 bfff 01f0 ff40 80bf ff01 f0ff 4080
> bfff 01f0 ff40 80bf ff01 f0ff 4080 bfff
> 01f0 ff40 80bf ff01 f0ff 4080 bfff 0000
> 0000 3f00 0001 f0ff 4080 bfff 01f0 ff40
> 80bf ff01 f0ff 4080 bfff 01f0 ff40 80bf
> ff01 f0ff 4080 bfff 01f0 ff40 80bf ff01
> f0ff 4080 bfff 01f0 ff40 80bf ff00 0000
> 003f 0000 01f0 ff40 80bf ff01 f0ff 4080
> bfff 01f0 ff40 80bf ff01 f0ff 4080 bfff
> 01f0 ff40 80bf ff01 f0ff 4080 bfff 01f0
> ff40 80bf ff01 f0ff 4080 bfff 0000 0000
> 3f00 0001 f0ff 4080 bfff 01f0 ff40 80bf
> ff01 f0ff 4080 bfff 01f0 ff40 80bf ff01
> f0ff 4080 bfff 01f0 ff40 80bf ff01 f0ff
> 4080 bfff 01f0 ff40 80bf ff00 0000 003f
> 0000 01f0 ff40 80bf ff01 f0ff 4080 bfff
> 01f0 ff40 80bf ff01 f0ff 4080 bfff 01f0
> ff40 80bf ff01 f0ff 4080 bfff 01f0 ff40
> 80bf ff01 f0ff 4080 bfff 0000 0000 3f00
> 0001 f0ff 4080 bfff 01f0 ff40 80bf ff01
> f0ff 4080 bfff 01f0 ff40 80bf ff01 f0ff
> 4080 bfff 01f0 ff40 80bf ff01 f0ff 4080
> bfff 01f0 ff40 80bf ff00 0000 003f 0000
> 01f0 ff40 80bf ff01 f0ff 4080 bfff 01f0
> ff40 80bf ff01 f0ff 4080 bfff 01f0 ff40
> 80bf ff01 f0ff 4080 bfff 01f0 ff40 80bf
> ff01 f0ff 4080 bfff 0000 0000 3f00 0001
> f0ff 4080 bfff 01f0 ff40 80bf ff01 f0ff
> 4080 bfff 01f0 ff40 80bf ff01 f0ff 4080
> bfff 01f0 ff40 80bf ff01 f0ff 4080 bfff
> 01f0 ff40 80bf ff00 0000 003f 0000 01f0
> ff40 80bf ff01 f0ff 4080 bfff 01f0 ff40
> 80bf ff01 f0ff 4080 bfff 01f0 ff40 80bf

Whereas raw rgba created with a command line conversion utility looks like this:

> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff
> 407f beff 407f beff 407f beff 407f beff

Just looking at the difference and from my reading, I can see that the rgba values do show up in the animation codec file, but aren’t available pixel by pixel. The 4080 bfff representing the solid color I’m recording for testing purposes is interspersed with other data, reflecting that the animation codec saves space by using the following technique:

“within a line, pixels are segmented into runs, the length of which is variable and signaled in the bitstream.”

Writing an a firmware version of the animation codec to read these .mov files seems like a pretty annoying process, whereas reading raw rgba off an SD card is fairly trivial. Would much rather do the latter in the interest of time. If TD doesn’t have a built-in option to create this I’ll just use a command line utility like ffmpeg.

I understand this is kind of an esoteric ask, but I thought it would be worth checking, since it would be nice if I could give other TD users a simple process for making content for my system, rather than having them need to interact with hacky command line stuff.

Not a movie file, but have you had a look at the .pcap format? I believe it does what you’re looking for - you’d likely need to make a wrapper for TD. I’ve done this in Max for LiDAR data and it works well, there’s no reason this wouldn’t work for RGB.

Hey @epalaima,

you could convert the TOP to a CHOP (TopTo CHOP) and save out the stream with the FileOut CHOP. The generated file is a bunch of columns and lot’s of numbers…

cheers
Markus

If you’re looking to save out RAW format RGB, then there is nothing that is prohibitive or inefficient about using sequence of images. The total file size will be just as large as a RAW format .mov, and will be just as fast to read as well. Renders out for movie CG regularly uses sequences of images for their ultra-high quality intermediate format output.

Animation codec is also a good option to reduce filesize, as the cost of CPU time. You can’t read the data directly, you need to decompress it, but it’ll turn into untouched RGB when decompressed properly. You’ll get the exact values out that you put into it.