I’m trying to do some color processing with OCIO, and having math rounding errors. The issue isn’t with OCIO specifically, this is more of a general question of how precision works in Touch Designer.
If I leave all the TOPs at 8-bit fixed, the CDL sliders work as expected. But I’d like to get higher precision to avoid banding. To simplify this example, I can actually remove OCIO and just use a gray ramp, measuring the min and max luminance with Analyze TOP. When the ramp is 8-bit fixed, the min and max are 0 and 1, as expected. But if I change the ramp to 16-bit fixed or 32-bit float, then I get rounding errors. Is there some way to avoid this?
For the Ramp that’s an artifact of how it’s generated. It’s taking a 0-1 ramp as a lookup, and pulling from that across the image. Since pixel centers are where sampling is done, the first sample is done a little bit inside of the Ramp, the last sample is done a little before the end of the ramp. In the simple case it seems weird, but when you start changing extend condition and moving the ramp phase around then things stay more correct this way.
TD works in whatever precision you have your TOPs set as. GLSL shaders internally always work in 32-bit floats, so all data is processed at 32-bit float during an operation, no matter the TOP format it is loaded from/saved to before and after the operation.
Likely a more specific example for your OCIO case is needed here.
