November 3rd, 2004, 06:33 PM
Precision of Numeric types in Delphi
Hi, I'm relatively new to Delphi, so be patient.
I'm processing a text file to feed a Firebird database. The text file contains 2 fields with monetary quantities, unformated separated by "|"'s.
I already finished the routine that parses the file and generates one with fixed lenght records to feed FB. My problem arises as I added a variable to sum the quantities along with parsing the file, and I can't get the same total as FB does. I'm using a double variable to hold the running sum, and using the Val procedure to convert from string to numeric inside Delphi. I already tested with a currency var and StrToCurr. In Delphi I get 86,710,857,594.49 and 86,710,857,594.39 with the currency var. In FB I get 86,711,048,279.58 as a decimal(15,2) sum, and 86,711,048,279.485 as a double precision. I assume Delphi is correct, but then how can I reproduce the right numbers in FB?
The source file contains 5.5 million records, so a 1 cent per record deviation produces a 500 thousand deviation on the end. Any sugestion?
Last edited by HereSomeHow; November 3rd, 2004 at 07:38 PM.
Reason: Correction of question
November 5th, 2004, 10:32 AM
Try using the Extended type and see what you get.
Up the Irons
What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
"Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
Down with Sharon Osbourne
"I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo