# rounding mystery

Build 56020
Windows 10
GTX 970M

Hi Derivative…

I’ve got a rounding quandary for you.

I’ve been playing with a few ideas for talking about Python, and I happened to be looking at the round() function.

When working with a value in a cell, I see the kinds of results that I expect:

value = 1.25677895123

round( float( value ) , 2 ) = 1.26
round( float( value ) , 4 ) = 1.2568
round( float( value ) , 5 ) = 1.25678

That all seems great to me.

If, however, I’m looking at the a CHOP, I see this:

op( ‘constant1’ )[ ‘chan1’ ] = 0.45
float( op( ‘constant1’ )[ ‘chan1’ ] ) = 0.44999998807907104
round( float( op( ‘constant1’ )[ ‘chan1’ ] ) , 2 ) = 0.45
round( float( op( ‘constant1’ )[ ‘chan1’ ] ) , 4 ) = 0.45
round( float( op( ‘constant1’ )[ ‘chan1’ ] ) , 8 ) = 0.44999999

It would seem that I’m missing something about the intricacies of CHOPs as they’re evaluated.

Am I not understanding what’s actually going on here, or is there a rounding error at play somewhere in here? I’m guessing that I’m missing something in terms of understanding how floats are calculated by CHOPs, but I figured it was worth asking.

I’ll attach a TOX as well.

Thanks in advance for any insight.

M
rounding.tox (686 Bytes)

Getting some other eyes on this, it seems like maybe trailing double zeros aren’t displayed… which would account for my confusion.

Whew!

Kindly disregard the post above.

Cheers.
M