Tuesday, May 3, 2011

how to get numbers to have precision of .05 ?

The following will ensure that any large numbers will only be precise to the hundredths place (related to this answer):

public function round( sc:Number ):Number
{
    sc = sc * 100;
    sc = Math.floor( sc );
    sc = sc / 100;

    return sc;
}

What is the optimal way to round my numbers to the precision of .05? Is there something clever to be done with bit-shifting to get this result? For example, I would like:

3.4566 = 3.45

3.04232 = 3.05

3.09 = 3.1

3.32 = 3.3

From stackoverflow
  • You could just change the *100 to *20 and /100 /20. But why stop there, when you can naturally generalize?

    double floor(double in, double precision) {
        return Math.floor(in/precision)*precision
    }
    //floor(1.07, 0.05) = 1.05
    
    Svante : This should be Math.round(), not Math.floor().
    Strilanc : His example used floor, so I used floor. Also notice I called the function floor to make it clear.
    Alnitak : To get consistent results for negative numbers you should also add 0.5 to the "in / precision" result passed to Math.floor()
    Strilanc : That would make it a rounding function, not a floor function. Floor rounds towards negative infinity. What you're thinking of is called truncation.
  • You could multiply by 20, round, then divide by 20.

    Edit: You will want to use Math.round() instead of Math.floor(), otherwise your test case 3.09 will turn into 3.05.

0 comments:

Post a Comment

Note: Only a member of this blog may post a comment.