Page 1 of 1

convert from scientific notation to number

Posted: Tue Jan 31, 2012 4:15 pm
by aitchon
I'm trying to get the long form of a number after adding 1 to it:


<$id="5835755762">
id:$id
<$maxid=(convert($id, 'double'))>
maxid:$maxid
<$maxid=($maxid+1)>
maxid:$maxid
<$convid=(convert($maxid, 'varchar'))>
convid:$convid

How can I get $convid to display 5835755763? It currently shows as 5.83576e+09.

convert from scientific notation to number

Posted: Tue Jan 31, 2012 4:48 pm
by mark
<fmt "%f" $convid> should do it.

convert from scientific notation to number

Posted: Tue Jan 31, 2012 4:51 pm
by John
Also if you are using Texis Version 6 you can use int64 instead of double:

<$id="5835755762">
id:$id

<$maxid=(convert($id, 'int64'))>
maxid:$maxid

<$maxid=($maxid+1)>
maxid:$maxid

convert from scientific notation to number

Posted: Tue Jan 31, 2012 4:52 pm
by aitchon
Adding this gave me:

5835760000.000000

It looks like converting it rounds it to the nearest thousand.

convert from scientific notation to number

Posted: Tue Jan 31, 2012 4:53 pm
by aitchon
I'm using 5.01 and would like to see if there's a workaround with this version.

convert from scientific notation to number

Posted: Tue Jan 31, 2012 5:44 pm
by Kai
Print it with <fmt "%.0f\n" $maxid> to suppress/round-off past the decimal.

convert from scientific notation to number

Posted: Tue Jan 31, 2012 6:06 pm
by aitchon
I believe the suppressing is happening in the convert($id, 'double').

convert from scientific notation to number

Posted: Wed Feb 01, 2012 10:03 am
by Kai
Doubles can have up to about 16 decimal digits of precision in the mantissa, so the convert varchar to double is not losing any digits; you'll see this if you print $maxid after the convert to double with <fmt "%.0f\n" $maxid>.

I'd have to see your full exact code for msg #4, but there's probably a convert double to varchar (perhaps implicitly) happening, which would use %g, which only uses 6 digits of precision by default -- that would cause the roundoff.

In your msg #1 example, just change:

<$convid=(convert($maxid, 'varchar'))>
convid:$convid

to:

<fmt "%.0f\n" $maxid>

and you'll see the full precision of $maxid.