#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Posts
    75
    Rep Power
    12

    VB 2010 fails at basic math???


    This is frustrating. I'm doing some really, really basic stuff, and getting a ridiculous result.

    Code:
    Private Function GetOverTimeHours() as double
        Dim TotalHours as double
        Dim RegularHours as double
    
        'a sql server connection is opened, and these two variables are set
    
        Return TotalHours-RegularHours
    End function
    Should be pretty straightforward, but I get some weird results. I had a case where TotalHours was 10.04, and RegularHours was 8. The function returned 2.0399999999999991. If I enter into my immediate window:
    Code:
    ?10.04-8
    I get the same result. Obviously, I can round the result, but I'm curious why this happens at all, so that I can hopefully avoid this in the future.
    ~~Macrado~~
  2. #2
  3. Transforming Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,240
    Rep Power
    9400
    Computers cannot represent all numbers perfectly: 10.04 on yours is probably stored as 10.0399999999999991. Subtract 8 from that and you get 2.0399999999999991.

    The solution is to round the result.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    4
    Rep Power
    0
    Notice, that even between 0 and 1 you have infinitely many numbers (e.g. 0.1, 0.01, 0.001, 0.0001, ...). It would be really wonderful world, if you could store them all in only 64 bits. Double variable store only rounding of a number.
    You can round numbers, like requinix said and it should work in most cases. However not always.
    For example, for checking equality of double1 and double2, you do should rather use this
    Code:
    abs(double1 - double2) < epsilon
    where epsilon is some small number (you can think about it as "the precision of equality").

    Still, floating-point numbers are pretty complex. If you want use them for something serious, you should do some research first.
    (you can start here http://www.cprogramming.com/tutorial/floating_point/understanding_floating_point.html )
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Posts
    75
    Rep Power
    12
    This is going to turn more into a rant than a request for help, so I'm sorry in advance.

    One cannot expect to have to round every number to a fixed number of decimal points after every mathematical operation. While I can certainly do it if necessary, the need for it in this case is completely unjustified.

    If I ask my calculator 10.04-8, it tells me 2.04, as it should. I would expect a sophisticated programming language to be as accurate.

    Here is another example that I ran into with the same program:

    Imagine you are writing a program that simply builds a receipt or an invoice. You write a function that accepts the selling price of an item, and how many of that item were sold.

    In my case, 3.2 quarts of oil were sold, at $1.56 per quart. So my function should return 3.2 * 1.56, which according to my calculator, is 4.992. According to vb, it is 4.9920000000000009. I can't go back and round every number I've ever performed basic math on.

    Again, I can type these values directly into my Immediate window and get the same results, so I can't imagine this is an error in my code.

    I appreciate the information as well as the suggestions, but something is clearly wrong.

    Last edited by macrado; December 10th, 2012 at 01:22 PM. Reason: clarity
    ~~Macrado~~
  8. #5
  9. Transforming Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,240
    Rep Power
    9400
    This is going to turn more into a rant than an answer, so I'm sorry in advance.

    What's "clearly wrong" is your understanding of computers. And of your calculator: when it calculates 3.2*1.56 it comes to the same answer your computer does. But whoever wrote the software for it actually understood the nature of computers and took appropriate precautions against showing you an answer that's a millionth of a millionth off. Because guess what the calculator does after it gets the answer? Rounds. Your calculator shows only so many digits so it rounds to an epsilon small enough to not impact the value displayed.

    So I'm sorry VB isn't the "sophisticated programming language" you expect it to be. I'm sorry the world of computing is more complicated than what you're used to seeing on a calculator. And I'm sorry that you have to go through the tremendous effort of adding a function call when you do floating-point arithmetic. You're right: something here is clearly wrong.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Posts
    75
    Rep Power
    12
    First off, chill the hell out. You seem to have misinterpreted my frustration as being with your response, which it was not. A personal attack is simply not cool.

    I've never worked with a programming language that made me handle rounding every bit of (floating point) arithmetic manually before. Therein lies my frustration.

    My initial question was why this occurs, not how I am supposed to fix it. That was explained nicely by number47, and his link was helpful. My second post was me expressing further frustration that VB doesn't handle this rounding itself, as every other language I've worked with does.

    So I'm sorry VB isn't the "sophisticated programming language" you expect it to be.
    Duh. I don't expect much from VB, but compare it to a calculator, and tell me which one has more sophistication. If I had a choice in which language to use, VB would not be it.

    I'm sorry the world of computing is more complicated than what you're used to seeing on a calculator.
    And I'm sorry for the bug that crawled into your *** and died.

    And I'm sorry that you have to go through the tremendous effort of adding a function call when you do floating-point arithmetic.
    I don't write the coding standards here, so I have no say in this. Ever hear of code review? With that said, no language I've ever worked with has required this, hence my confusion and frustration.

    So yeah, thank you Number47 for being helpful, and requinix thanks for the laugh at your misunderstanding :-)
    Last edited by macrado; December 10th, 2012 at 04:12 PM.
    ~~Macrado~~
  12. #7
  13. No Profile Picture
    Still Learning
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Location
    Montreal, Canada
    Posts
    55
    Rep Power
    39

    Programming Languages and Real numbers


    I would like to comment on the fact that the OP said none of the previous languages exposed this issue to him before.

    It is not uncommon that languages have internal default settings for the display of real numbers. It takes an explicit format to extend it past the default.

    This default issue has wider consequences. What else do we not know about the behavior of our programming languages.

    I discovered years ago the compiler options that have a primary purpose sometimes a secondary effect. In my case turning on an optimization gave different results. No code change, just that the two executable programs ran differently with the different optimization setting.
  14. #8
  15. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    4
    Rep Power
    0
    Woow, slow down guys. (btw how do you spell woow in English?)
    macrado, you complained at something which is natural for computer beings, and it has looked like demanding of something more. You know, it can be a little bit irritating. Thousands of computer scientists try hard to find the best solution, and someone comes and says "Oh, it's so stupid, they should do something more".

    admiraln, the optimization can change the order of evaluation, so if something depends on the order the result can be different. But, if it is another matter it can be a bug in the compiler. Or optimization has some code requirements ("you must use this", "you mustn't use this", ...).

    Oh, and sorry for smarassing, but you know, I'm so wonderful and everything, so ... I couldn't resist.

    That was explained nicely by number47, and his link
    I'm a girl. A pat and one pack of puffs are owed ^-^
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Posts
    75
    Rep Power
    12
    Sorry for all of my frustration guys. We just switched over to VB 2010 for several of our projects here, and it has been frustrating trying to update our code. On top of that, management has suddenly become extremely nitpicky about our coding standards (No using this function, no using that function, variable names will follow this specific format, etc). Finding that some basic operations were not returning the results I expected just pushed me over the edge, I guess.

    As I mentioned, other languages I've worked with, including earlier versions of vb, have always done the rounding on floating point variables themselves, so I was freaking out about what was going on. The closest I've come to this is some rounding issues while storing single point floats into a database.

    I really do appreciate the help that was given even if some of it had to come in the form of some tough love.

    I'm a girl. A pat and one pack of puffs are owed ^-^
    Nonsense. Everyone knows there are no girls on the internet. Thank you again for your help!
    ~~Macrado~~
  18. #10
  19. ASP.Net MVP
    Devshed Specialist (4000 - 4499 posts)

    Join Date
    Aug 2003
    Location
    WI
    Posts
    4,378
    Rep Power
    1511
    It's called IEE 754 people, and pretty much every programming language uses it. I'm having a hard time thinking of one that doesn't.

    That's what's kind of silly here: why call out VB.Net on this, when nothing else you try will be any different in spite of what you say about prior languages doing this for you?
    Primary Forum: .Net Development
    Holy cow, I'm now an ASP.Net MVP!

    [Moving to ASP.Net] | [.Net Dos and Don't for VB6 Programmers]

    http://twitter.com/jcoehoorn
  20. #11
  21. No Profile Picture
    Still Learning
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Location
    Montreal, Canada
    Posts
    55
    Rep Power
    39
    I am curious does the fact CPU hardware or a software library use iEEE 754 force a programming language to display a certain precision. I suspect not. The under lying hardware my be running double precision but the default display is say only 10 significant digits.
    In this case you would not see the details of the implementation.
  22. #12
  23. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2013
    Posts
    3
    Rep Power
    0

    Wrong Data Type


    For what you're expecting, I think you really wanted to use "decimal" instead of double. Decimal will add and subtract in the manner you expect.

IMN logo majestic logo threadwatch logo seochat tools logo