Friday, May 6, 2011

Float calculation errors

in C#

float ratio = 185 / srcRect.Width;

where srcRect.Width = 400

returns 0 where it should return 0.45...

min value for float is -3.40282347E+38 so how is this possible?

From stackoverflow
  • You are performing an integer division and hence 185/srcRect.Width will return 0. You will need to change the expression to

    float ratio = 185.0f / srcRect.Width;
    
    gimel : float ratio = 185.0 / srcRect.Width;
    plinth : gimel - that won't work as written - the RHS is type double, which can't be assigned to a float without a cast.
  • 185 is an integer literal, and so the division performed is integer division.

    To perform a float division, use 185.0 or 185.0f instead (the former is a double, the latter is float)

  • The "f" or "d" suffixes may be added to literal numbers to turn them into floats or doubles respectively.

    Like so:

    float ratio = 185f / srcRect.Width;
    

    note that "185.0" is actually a double unless it also has the "f" suffix.

0 comments:

Post a Comment

Note: Only a member of this blog may post a comment.