Rounding by MvEVAL?

[Kent Multer]

Hi follks,

I've been working on a module that computes prices for custom draperies, doing quite a bit of math based on the customer's attribute selections for size, fabric type, etc. I've seen some results that seem a little off. I understand that some of this is to be expected when working on a computer, because of rounding errors in the very last digit. For instance, I might see a result of 13.99999999 or 14.00000001 when the correct answer is 14.

The module displays a lot of intermediate results and other data on the screen for debugging. Recently,I had a case where an intermediate result was displayed on the screen as 13.75, but the next step in the math came out with a result that was too high. This, plus a couple of additional tests, seems to suggest that the actual value of the variable was 13.75000001, but that MvEVAL is hard-coded to round off a very small overflow. Is that correct? Is there some automatic rounding built into MvEVAL (and anywhere else in the VM)?

Thanks --

[burch]

When converting floating point numbers to strings (necessary for output, in your case), we use a widely available function called dtoa() to do the conversion. You can see the source code and read some comments from the author here: http://www.netlib.org/fp/dtoa.c

[Kent Multer]

Thanks, Burch, but that file is almost 5000 lines long, and doesn't seem to contain any concise description of what it actually does. I tried a couple of Google searches, but didn't find any docs on this. I get the impression that the answer to my question is, Yes, MvEVAL does number-to-string conversion that involves some rounding, so that a value of 13.75000001 might be displayed as 13.75. Is that correct?

Thanks --