precision of float

[jren]

Hi,

I have two questions related to precision of float:

1.
In my SKILL code:
grid1 = 0.5
fprintf(stdout "grid1 = %.20f\n" grid1)
grid2 = 0.4
fprintf(stdout "grid2 = %.20f\n" grid2)
Output in CIW:
grid1 = 0.50000000000000000000
grid2 = 0.40000000000000002220

How can I get 0.4 = 0.4000000000000000 as 0.5?

2.
I tried to use function techSetPrecision(), but I am confused by the
following test:
techSetPrecision(2)
grid3 = 0.5553567
grid3 = grid3 * 1.0
fprintf(stdout "grid3 = %.20f\n" grid3)
In the CIW:
grid3 = 0.55535670000000003643
What I assume is grid3 should become 0.5600000 since I set precision
of number of digits in float as 2. Can anyone help me better
understand function techSetPrecision?

Thanks!

Jren

 

[Jean-Marc Bourguet]

jren <ddsweet@gmail.com> writes:

In my SKILL code:
grid1 = 0.5
fprintf(stdout "grid1 = %.20f\n" grid1)
grid2 = 0.4
fprintf(stdout "grid2 = %.20f\n" grid2)
Output in CIW:
grid1 = 0.50000000000000000000
grid2 = 0.40000000000000002220

How can I get 0.4 = 0.4000000000000000 as 0.5?

You can't. Floating point is a pain to work with. Just like you can't
represent 1/3 exactly in decimal with a fixed number of digits, you
can't represent 0.4 exactly in common floating point format as they are
using a binary representation (they are able to represent only a subset
of fractions with a power of 2 as denominator, so 0.5 which is 1/2 can
be represented exactly, but 0.4 must be represented as an approximation)

Yours,

--
Jean-Marc

 

[jren]

On Sep 7, 3:34 am, Jean-Marc Bourguet <j...@bourguet.org> wrote:

jren <ddsw...@gmail.com> writes:
In my SKILL code:
   grid1 = 0.5
   fprintf(stdout "grid1 = %.20f\n" grid1)
   grid2 = 0.4
   fprintf(stdout "grid2 = %.20f\n" grid2)
Output in CIW:
        grid1 = 0.50000000000000000000
        grid2 = 0.40000000000000002220

How can I get 0.4 = 0.4000000000000000 as 0.5?

You can't.  Floating point is a pain to work with.  Just like you can't
represent 1/3 exactly in decimal with a fixed number of digits, you
can't represent 0.4 exactly in common floating point format as they are
using a binary representation (they are able to represent only a subset
of fractions with a power of 2 as denominator, so 0.5 which is 1/2 can
be represented exactly, but 0.4 must be represented as an approximation)

Yours,

--
Jean-Marc

Thanks a lot for your explanation.

Regards!

Jren

 

[Andrew Beckett]

jren wrote, on 09/07/11 12:06:

On Sep 7, 3:34 am, Jean-Marc Bourguet<j...@bourguet.org> wrote:
jren<ddsw...@gmail.com> writes:
In my SKILL code:
grid1 = 0.5
fprintf(stdout "grid1 = %.20f\n" grid1)
grid2 = 0.4
fprintf(stdout "grid2 = %.20f\n" grid2)
Output in CIW:
grid1 = 0.50000000000000000000
grid2 = 0.40000000000000002220

How can I get 0.4 = 0.4000000000000000 as 0.5?

You can't. Floating point is a pain to work with. Just like you can't
represent 1/3 exactly in decimal with a fixed number of digits, you
can't represent 0.4 exactly in common floating point format as they are
using a binary representation (they are able to represent only a subset
of fractions with a power of 2 as denominator, so 0.5 which is 1/2 can
be represented exactly, but 0.4 must be represented as an approximation)

Yours,

--
Jean-Marc

Thanks a lot for your explanation.

Regards!

Jren

Note also that this not just a SKILL issue. You'd have exactly the same problem
with most programming languages (certainly any that use IEEE floating point
representation, including C).

Andrew.