Encoding Type used to store input

[mark]

No. contenttypeparam is the type specified in the downloaded page. charsettxt is what the extracted text is. It could be different.

[aitchon]

If I detect that <urltext> produces invalid UTF-8 data, is there a way to replace all invalid UTF-8 characters with a valid character like a space?

[mark]

If you can detect them you can replace them. See <sandr>.

[aitchon]

I used the procedure in message#11 to detect for existence of invalid UTF-8 characters, but it seems that I found some text that was produced from urltext used the following escape sequence and wasn't detected:

&#3;

Should I just replace anything that starts with &# and ends with ;?

[mark]

Sounds reasonable.