As a follow-up to a post I wrote recently about JavaScript maths errors here's another. This time it's about the same issue in Lotus Notes @Formula-based maths.
See Exhibit A below:
As mentioned previously it's nothing to do with the language used. It's all down to the hardware.
Always remember to @Round the result of any mathematics, like so:
Written by Jake Howlett on Thu 1 Aug 2013
The most recent comments added:
Skip to the comments or add your own.
You can subscribe to an individual RSS feed of comments on this entry.
CodeStore is all about web development. Concentrating on Lotus Domino, ASP.NET, Flex, SharePoint and all things internet.
Your host is Jake Howlett who runs his own web development company called Rockall Design and is always on the lookout for new and interesting work to do.
You can find me on Twitter and on Linked In.
Read more about this site »
I don't know if it's still the case but it used to be an issue with rounding that @Formula would round one way and LotusScript would round t'uther!
This came to light on an e-commerce site where the basket used @Formula on screen but the checkout processing was done by a LS lib behind the scenes and they always had differing amounts!
Reply
Scientific vs Accounting rounding. I'm not sure which one @formula uses by default but it's the reason the Currency datatype exists in LS.
Reply
Show the rest of this thread
Yep, floating point is a pain, but nothing new. We've had 'irrational numbers' for centuries! 1/3 as a decimal = ? 0.33333... or root 2 = 1.414213..., natural log anybody? Binary just does a similar approximation in base 2.
The main issue that people seem to fail to deal with is the difference between the internal binary representation and the 'displayed value' that often has an implied internal rounding process. Where this gets nasty is when you total up long lists of numbers and get a total which is slightly different to whats displayed. This is what 'Decimal/Money' types are for in other databases. They are 'scaled integers' and don't use floating point.
Either get familiar with fix/int/mod/round or "\" and apply judicious rounding or scale your own integers. For example, %percents are often calculated as 0-100 and not 0.00-1.00.
Its this sort of stuff that sorts the men from the boys..!
Reply
1/3 is not an irrational number. :-)
Reply
NoNoNo... You don't round, you tranfer that bit to your offshore account.
Reply
There's something wrong also in LSS:
Don't know if in 8.5.3 this was fixed...
Reply
Hey Jake, there's something wrong also on your blog parsing urls :)
Reply