charsetconv

aitchon · Post by **aitchon** » Wed Jun 25, 2008 12:27 pm

I'm using <urlutil charsetconv> to convert some text from gb2312 to UTF-8. Here is the resulting text:

回复：中国-四川=? ****

But when I write that text to an XML file, I get an invalid character error when trying to open that XML file in IE.

Post by **John** » Wed Jun 25, 2008 3:49 pm

The  and  are not valid XML characters. What was in the original text?

aitchon · Post by **aitchon** » Wed Jun 25, 2008 4:46 pm

Why would <urlutil charsetconv> return invalid XML if I'm using it to convert to UTF-8. Is it possible that iconv is failing? If so, how would I go about making sure that the returned text doesn't contain invalid XML characters?

Post by **Kai** » Wed Jun 25, 2008 4:58 pm

It's possible that iconv is failing. Are there any putmsgs when it is run? To validate purported UTF-8 output, you could decode and re-encode it with <strfmt>:

<strfmt "%!hV" $ret>
<strfmt "%hV $ret>

Then any invalid UTF-8 chars would be mapped to `?'. The `h' subflag encodes out-of-ISO-8859-1-range chars as HTML, to prevent losses.

Note that some browsers may still complain about certain *valid* UTF-8 chars, such as U+0080 through U+009F.

aitchon · Post by **aitchon** » Wed Jun 25, 2008 5:10 pm

I turned on putmsgs and see this:

Cannot completely convert charset gb2312 to UTF-8 via converter "c:\morph3\etc\iconv" -f gb2312 -t UTF-8 -c: returned exit code 1 in the function httransbuf

charsetconv still returns a value. So I shouldn't assume that this return value will always be valid UTF-8? Should I look for this error first?

aitchon · Post by **aitchon** » Wed Jun 25, 2008 5:27 pm

Here's my code:
<$var="»ظ´£ºÖйú-ËĴ¨=? ??****">
<$charset="gb2312">
<urlutil charsetconv $var $charset "UTF-8">
<if $ret ne "">
<strfmt "%!hV" $ret>
<strfmt "%hV" $ret>
<strfmt "%H" $var>
<$var = $ret>
</if>

For this specific call, I don't get the error in putmsgs mentioned above. But it still produces invalid UTF-8.

Post by **John** » Wed Jun 25, 2008 6:01 pm

The ?? are the problematic characters, they get converted to valid UTF-8, but not valid XML as they are control characters, which are not allowed in XML even though they are valid in UTF-8.

aitchon · Post by **aitchon** » Wed Jun 25, 2008 6:55 pm

I assume I should try to detect and remove these control characters before trying to convert to UTF-8. How would I go about detecting these control characters?

Post by **John** » Thu Jun 26, 2008 10:19 am

You could sandr them out of the source, before the charsetconv e.g.

<$s="[\x00-\x08\x0b\x0c\x0e-\x1f]+">
<$r=" ">
<sandr $s $r $var>
<$var=$ret>

to replace strings of control characters with spaces. It will leave tabs, CR and LF alone as they are acceptable.

aitchon · Post by **aitchon** » Wed Jul 30, 2008 4:39 pm

To avoid any errors like those mentioned in message #5 when converting from gb2312 to utf8, are there any other alternate programs that will convert from gb2312 to utf8?