Zsh Mailing List Archive
Messages sorted by: Reverse Date, Date, Thread, Author

Re: arithmetic anomaly



On Thu, Mar 25, 2021 at 11:44 PM Ray Andrews <rayandrews@xxxxxxxxxxx> wrote:
On 2021-03-25 1:40 p.m., Lawrence Velázquez wrote:
>
> The bad news is that *all* your figures -- your numerators, Brett's
> numerators, and the denominators -- are actually incorrect for
> level=15 and higher.
Sheesh, what gives?  Seems no accuracy beyond 15 digits?

Indeed. 64-bit floating point numbers usually have 15.95 decimal digits of precision. Depending on the compiler and how you write your code, you might get higher precision but in practice many projects turn this off because having borderline non-deterministic results of computation sucks.

This isn't specific to zsh. Floating numbers are nowadays the same everywhere (IEEE 754).

Roman.


Messages sorted by: Reverse Date, Date, Thread, Author