*X-seq*: zsh-users 5287*From*: David Kågedal <davidk@xxxxxxxxxxxxxx>*To*: Patrick Aussems <zyk@xxxxxxxxxxxxxx>, Vincent Lefevre <vincent@xxxxxxxxxx>*Subject*: Re: Floating point calculus error...*Date*: Sun, 25 Aug 2002 18:32:38 +0200*Cc*: zsh-users@xxxxxxxxxxxxxx*In-reply-to*: <20020825155349.GD6783@xxxxxxxxxxxxxx> (Vincent Lefevre's message of "Sun, 25 Aug 2002 17:53:50 +0200")*Mailing-list*: contact zsh-users-help@xxxxxxxxxx; run by ezmlm*References*: <1030288941.517.8.camel@Amok> <20020825155349.GD6783@xxxxxxxxxxxxxx>

Vincent Lefevre <vincent@xxxxxxxxxx> writes: > On Sun, Aug 25, 2002 at 17:22:20 +0200, Patrick Aussems wrote: >> I was doing some shell scripts that were supposed to add CPU usages from >> the ps output... But I got puzzled when I got an answer with a 17th >> decimal while adding numbers with only 1 decimal... Strange isn't it? > > The numbers are internally represented in binary and 0.4 isn't exactly > representable in binary... But here, this is a problem with zsh, which > gives an additional decimal digit. You can see with: > > $ echo $[0.4] > 0.40000000000000002 > > I think that it should give the minimum number of decimals in such a way > that when converting back to binary, one should get the same result. But it does. $ echo $[.40000000000000002] 0.40000000000000002 The point is, as you say, that 0.4 can't be represented as an exact binary floating-point number, no matter how large the representation is. Just like 1/3 can't be represented as a decimal floating-point number. -- David Kågedal

**Follow-Ups**:**Re: Floating point calculus error...***From:*Vincent Lefevre

**References**:**Floating point calculus error...***From:*Patrick Aussems

**Re: Floating point calculus error...***From:*Vincent Lefevre

Messages sorted by: Reverse Date, Date, Thread, Author