convert from scientific notation to number

Post Reply
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

convert from scientific notation to number

Post by aitchon »

I'm trying to get the long form of a number after adding 1 to it:


<$id="5835755762">
id:$id
<$maxid=(convert($id, 'double'))>
maxid:$maxid
<$maxid=($maxid+1)>
maxid:$maxid
<$convid=(convert($maxid, 'varchar'))>
convid:$convid

How can I get $convid to display 5835755763? It currently shows as 5.83576e+09.
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

convert from scientific notation to number

Post by mark »

<fmt "%f" $convid> should do it.
User avatar
John
Site Admin
Posts: 2597
Joined: Mon Apr 24, 2000 3:18 pm
Location: Cleveland, OH
Contact:

convert from scientific notation to number

Post by John »

Also if you are using Texis Version 6 you can use int64 instead of double:

<$id="5835755762">
id:$id

<$maxid=(convert($id, 'int64'))>
maxid:$maxid

<$maxid=($maxid+1)>
maxid:$maxid
John Turnbull
Thunderstone Software
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

convert from scientific notation to number

Post by aitchon »

Adding this gave me:

5835760000.000000

It looks like converting it rounds it to the nearest thousand.
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

convert from scientific notation to number

Post by aitchon »

I'm using 5.01 and would like to see if there's a workaround with this version.
User avatar
Kai
Site Admin
Posts: 1271
Joined: Tue Apr 25, 2000 1:27 pm

convert from scientific notation to number

Post by Kai »

Print it with <fmt "%.0f\n" $maxid> to suppress/round-off past the decimal.
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

convert from scientific notation to number

Post by aitchon »

I believe the suppressing is happening in the convert($id, 'double').
User avatar
Kai
Site Admin
Posts: 1271
Joined: Tue Apr 25, 2000 1:27 pm

convert from scientific notation to number

Post by Kai »

Doubles can have up to about 16 decimal digits of precision in the mantissa, so the convert varchar to double is not losing any digits; you'll see this if you print $maxid after the convert to double with <fmt "%.0f\n" $maxid>.

I'd have to see your full exact code for msg #4, but there's probably a convert double to varchar (perhaps implicitly) happening, which would use %g, which only uses 6 digits of precision by default -- that would cause the roundoff.

In your msg #1 example, just change:

<$convid=(convert($maxid, 'varchar'))>
convid:$convid

to:

<fmt "%.0f\n" $maxid>

and you'll see the full precision of $maxid.
Post Reply