WinBear
Joined: 06 Jan 2006
Posts: 1
Location: Denton, Tx
|
 Posted: Fri Jan 06, 2006 2:08 am Post subject: Math errors in Lua Player for Windows and crashes PSP |
 |
|
I've been working on a Lua Player game. I'm requiring compatibility with the Microsoft random number generator/seed code, which I have. This particular code depends on the behavior of "long integers" wrapping when exceeding the capacity of 32 bit numbers. Lua Player uses floating point and I read some of the arguments about that. I'm fairly confident I can work around that feature and make the numbers behave the same way. However, Lua Player for Windows introduces a rounding error which I cannot work around.
Here's what I'm seeing:
Using the following formula
x = 2745024 * 214013 + 2531011
Lua Player for Windows gives a result of x = 587473354752
Windows Calculator says the result should be x = 587473352323
The difference of 2429 looks a little odd until you see the differences in hexadecimal.
In Lua Player for Windows, x = 0x88C8240000
In Calculator for Windows, x = 0x88C823F683
It's rounding up by 2 to the 16th power.
I went to try this on the PSP and Lua Player for the PSP completely crashes trying to do the math calculation, shutting down the PSP after about 10 seconds.
_________________
Tommy in Denton |
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
