Thunderstone Support Forums

Using exec, how would I go about using iconv to convert text in a character set to UTF-8?

<exec bin iconv -f $sourcecharset -t "UTF-8"><fmt "%s" $datatoconvert></exec>
<$converteddata=$ret>

See also
http://www.gnu.org/software/libiconv/do ... onv.1.html

Would this be a more effective way than using <fmt "%hhV" $str> when trying to output text to an XML file?

<fmt> is builtin to Vortex and thus is faster than <exec>ing a separate iconv process. Also, you'd need to HTML-escape the iconv output (for XML) after the <exec> with <fmt "%H">, whereas <fmt "%hhV"> does that already. (But note that "%hhV" assumes the *input* has HTML-escaped its ampersands, and must be ISO-8859-1.)

A more generic -- and just as fast -- way to convert charsets is with:

<urlutil charsetconv $datatoconvert $sourcecharset "UTF-8">
<strfmt "%H" $ret> <!-- HTML-escape for XML -->
<$converteddata = $ret>

which will handle any builtin or iconv charset. It will use the same (internal/fast) routines as <fmt> if the charset pair can be handled that way, otherwise it <exec>s iconv. Note the <strfmt> for HTML-escaping for XML.

So if I already know that text is already encoded in UTF-8, would only have to <strfmt "%H" $var> to output that variable to XML?

Correct, %H does the HTML escaping of things like & and <

If the $sourcecharset is equal to "Unknown", is there any attempt to try to convert the data to UTF-8? Or should I not even bother trying to call urlutil charsetconv?

Any charset that is not known internally is punted to an exec'd iconv process. So $sourcecharset of "Unknown" would indeed exec an iconv; you should not bother calling <urlutil charsetconv> then.

I'm using <urlutil charsetconv> to convert a piece of text to UTF-8 from a page that's encoded in ISO-8859-1. It seems that it's having trouble converting the some characters in this string:

gameplay variety to the series — players

is translated to:

gameplay variety to the series â&#128;&#148; players

Is this the correct conversion to UTF-8? The translated characters do not seem to be valid UTF-8 characters.

It looks like that string was already in UTF-8 (an em dash U+2014); converting a UTF-8 page as if it were ISO-8859-1 will indeed result in incorrect results. Perhaps the page's charset was incorrectly labelled?