Fetching HTML with foreign characters, like ê,ñ,ç

gerry.odea · Post by **gerry.odea** » Fri Dec 05, 2008 8:55 am

I'm running into a problem when I fetch html that has foreign characters in the url strings on the html page.

For instance if I try to bring in a page that has these urls below, thunderstone will drop the foreign characters from the urls but not from the description.

<a href="/Top/World/Português">Português</a>
<a href="/Top/World/Français">Français</a>
<a href="/Top/World/Español">Español</a>

When I get them back from a fetch they look like this:

<a href="/Top/World/Portugus">Português</a>
<a href="/Top/World/Franais">Français</a>
<a href="/Top/World/Espaol">Español</a>

This is the fetch I am using:

<a name=DIRECTOR>
<if $urls like "/Top">
<fetch PARALLEL $urls>
<sandr '/Top' '/texis/open/geekie\?urls=/Top' $ret>
<$html=$ret>
<send $html>
<flush></fetch>
</if>
</a>

Post by **John** » Fri Dec 05, 2008 10:05 am

How is the page encoded to begin with? The URL should probably be URL encoded in any case. Is there an example URL we can fetch?

gerry.odea · Post by **gerry.odea** » Fri Dec 05, 2008 10:52 am

It's not encoded. It's just like this:

<table width=98% cellpadding=2 cellspacing=0>
<tr><td nowrap>
<td width=40% valign="top"><a href="/Top/World/Afrikaans">Afrikaans</a> (581) <a href="/Top/World/Arabic">Arabic</a> (7103) <a href="/Top/World/Armenian">Armenian</a> (896) <a href="/Top/World/Asturianu">Asturianu</a> (95) <a href="/Top/World/Azerbaijani">Azerbaijani</a> (594) <a href="/Top/World/BahasaIndonesia">Bahasa Indonesia</a> (2620) <a href="/Top/World/BahasaMelayu">Bahasa Melayu</a> (601) <a href="/Top/World/Bangla">Bangla</a> (29) <a href="/Top/World/Belarusian">Belarusian</a> (222) <a href="/Top/World/Bosanski">Bosanski</a> (1770) <a href="/Top/World/Brezhoneg">Brezhoneg</a> (241) <a href="/Top/World/Bulgarian">Bulgarian</a> (5298) <a href="/Top/World/Català">Català</a> (46684) <a href="/Top/World/ChineseSimplified">Chinese Simplified</a> (61518) <a href="/Top/World/ChineseTraditional">Chinese Traditional</a> (16392) <a href="/Top/World/Cymraeg">Cymraeg</a> (628) <a href="/Top/World/Dansk">Dansk</a> (68284) <a href="/Top/World/Deutsch">Deutsch</a> (952435) <a href="/Top/World/Eesti">Eesti</a> (1307) <a href="/Top/World/Español">Español</a> (203497) <a href="/Top/World/Esperanto">Esperanto</a> (3841) <a href="/Top/World/Euskara">Euskara</a> (2237) <a href="/Top/World/Français">Français</a> (320926) <a href="/Top/World/Frysk">Frysk</a> (37) <a href="/Top/World/Furlan">Furlan</a> (170) <a href="/Top/World/Føroyskt">Føroyskt</a> (74) <a href="/Top/World/Gaeilge">Gaeilge</a> (112) <a href="/Top/World/Galego">Galego</a> (1815) <a href="/Top/World/Greek">Greek</a> (4071) <a href="/Top/World/Gujarati">Gujarati</a> (66) <a href="/Top/World/Gàidhlig">Gàidhlig</a> (199) <a href="/Top/World/Hebrew">Hebrew</a> (6836) <a href="/Top/World/Hindi">Hindi</a> (1005) <a href="/Top/World/Hrvatski">Hrvatski</a> (5695) <a href="/Top/World/Interlingua">Interlingua</a> (91) <a href="/Top/World/Italiano">Italiano</a> (224607) <a href="/Top/World/Japanese">Japanese</a> (192104) <a href="/Top/World/Kannada">Kannada</a> (105) <a href="/Top/World/Kaszëbsczi">Kaszëbsczi</a> (37) <a href="/Top/World/Kazakh">Kazakh</a> (147) </td>
<td width=60% valign="top"><a href="/Top/World/Kiswahili">Kiswahili</a> (47) <a href="/Top/World/Korean">Korean</a> (7226) <a href="/Top/World/Kurdî">Kurdî</a> (584) <a href="/Top/World/Latviski">Latviski</a> (4142) <a href="/Top/World/Lietuvių">Lietuvių</a> (6684) <a href="/Top/World/LinguaLatina">Lingua Latina</a> (89) <a href="/Top/World/Lëtzebuergesch">Lëtzebuergesch</a> (30) <a href="/Top/World/Magyar">Magyar</a> (9378) <a href="/Top/World/Makedonski">Makedonski</a> (237) <a href="/Top/World/Marathi">Marathi</a> (45) <a href="/Top/World/Nederlands">Nederlands</a> (115946) <a href="/Top/World/Norsk">Norsk</a> (17028) <a href="/Top/World/Occitan">Occitan</a> (106) <a href="/Top/World/Ossetian">Ossetian</a> (24) <a href="/Top/World/Persian">Persian</a> (1546) <a href="/Top/World/Polska">Polska</a> (87472) <a href="/Top/World/Português">Português</a> (28375) <a href="/Top/World/Punjabi">Punjabi</a> (66) <a href="/Top/World/Română">Română</a> (18450) <a href="/Top/World/Rumantsch">Rumantsch</a> (86) <a href="/Top/World/Russian">Russian</a> (70223) <a href="/Top/World/Sardu">Sardu</a> (308) <a href="/Top/World/Shqip">Shqip</a> (432) <a href="/Top/World/Sicilianu">Sicilianu</a> (38) <a href="/Top/World/Slovensko">Slovensko</a> (953) <a href="/Top/World/Slovensky">Slovensky</a> (3386) <a href="/Top/World/Srpski">Srpski</a> (2992) <a href="/Top/World/Suomi">Suomi</a> (11204) <a href="/Top/World/Svenska">Svenska</a> (46678) <a href="/Top/World/Tagalog">Tagalog</a> (532) <a href="/Top/World/Taiwanese">Taiwanese</a> (138) <a href="/Top/World/Tamil">Tamil</a> (285) <a href="/Top/World/Tatarça">Tatarça</a> (91) <a href="/Top/World/Telugu">Telugu</a> (166) <a href="/Top/World/Thai">Thai</a> (2098) <a href="/Top/World/Türkçe">Türkçe</a> (1113711) <a href="/Top/World/Ukrainian">Ukrainian</a> (5531) <a href="/Top/World/Vietnamese">Vietnamese</a> (892) <a href="/Top/World/slenska">Íslenska</a> (512) <a href="/Top/World/Česky">Česky</a> (25621) </td>
</tr><tr><td height="10"></td></tr></td></table>

Post by **mark** » Fri Dec 05, 2008 12:09 pm

I'm not able to replicate what you describe using what you've posted. Perhaps there's some webserver or browser interaction going on? Exactly where are you seeing the missing characters? Try writing the result of the fetch to a file and the result of the sandr to another file and compare those to the original.

What version of texis are you using?

gerry.odea · Post by **gerry.odea** » Fri Dec 05, 2008 12:20 pm

There is no difference on the fetch, maybe it has to do with the old version of Texis that I have?

I have:
Texis Web Script (Vortex) Copyright © 1996-1999 Thunderstone - EPI, Inc.
Commercial Version 2.6.929642470 of Jun 17, 1999 (i686-unknown-linux2.2.5)

Is there a work around for older versions of Texis?

Post by **mark** » Fri Dec 05, 2008 1:43 pm

I doubt I can resurrect anything that old to develop workarounds. Are you saying that sandr is causing the difference? We need to find exactly where the problem is and figure what's happening. At what point does the change first appear?

gerry.odea · Post by **gerry.odea** » Fri Dec 05, 2008 2:14 pm

The change appears with the Fetch, not the sandr, if it was just sandr that would be easy. I removed the sandr and it was occuring still. The Fetch seems to be not be recognizing the characters such as ê,ñ,ç and is just dropping them off. Is there any command I can put in to stop that?

Post by **mark** » Fri Dec 05, 2008 4:32 pm

Are you seeing the difference in the browser or in a file saved from the script?

<fetch PARALLEL $urls>
<$html=$ret>
<write /tmp/sample.html><fmt "%s" $html></write>
<fmt "%s" $html>
</fetch>

Post by **mark** » Fri Dec 05, 2008 4:35 pm

Do you have any urlcp calls? Especially any reparent options?