Accented Noise words and synonyms

[edev]

Hi,

it seems like French accented characters are giving me more trouble than anything else. I got the equivalence file to work (created a customized "eqvsusr" file), but when I include French words with accented characters in the list the backref.exe runs fine, but when the search does not pick up the equivalence.

For example, if I have in my eqvsusr.lst

montréal,montreal
grâce,grace,honor

the results will not include any of the equivalent words. I typed in the accented é,â in wordpad by using "alt+131" and "alt+133".

I also modified the $noiselist included in the search script, but it does not pick anything with accents. For example, I want the search script to treat "à" as a noise word. I tried
<$noiselist="à" "&agrave;" "\xC3\xA0" "\xE0">
<apicp noise $noiselist>

and it does not work.

Is there a special format for accented characters to be included in thesaurus and noiselist? Thanks.

[edev]

I'm confused - what are the wordc and langc settings and where can I find them? Thanks.

[mark]

http://thunderstone.master.com/texis/ma ... ml?q=wordc

Also make sure your equivs include the character sets being used.

[edev]

Thanks guys. I'm not sure if accented characters would be included in "\alpha", but I put the SQL statements in the <search> function:

<A NAME=search>
<local savenext saveindexcount>

<sql "set wordc='[\alnum]'"></sql>
<sql "set langc='[\alnum]'"></sql>
...

and I also included them in the <init> function. None of them worked. I'm not sure if the accented characters are a part of alpha characters, or if I had placed the SQL statements in the wrong place. I'm using UTF-8 as searching character sets. Thanks again for any help you could provide...

[mark]

They should be in the fpar function like the other set's. You've made them more restrictive not less. Put space and - in langc as in the default case. Also try adding all the high bits to them both.

<sql "set wordc='[\alnum\x80-\xff'']'"></sql>
<sql "set langc='[\alnum\x80-\xff'' \-]'"></sql>

Make sure you include the utf-8 version of montréal in your equiv.

[edev]

Thanks Mark!

I changed the settings to fpar, but I'm not sure what you mean by including the utf-8 version in my equiv. In my eqvsusr.lst, I have for example:

montréal,Montreal,Montréal,montreal

the accents are entered by wordpad using the alt+130 key, by including the utf-8 version do I have to translate the English words to utf-8 code, such as:

montr\xC3\A9al,montreal

or every single letter in utf-8 code?

[edev]

Another question, after including the sql statements in my fpar function, I have my noiselist set to exclude the noise word "à", so I have

<$noiselist = "a" "à" "\xE0" "&Agrave;" "\xC3\xA0" ...>

to include every possible format of the letter à, but when I do a search it still does not filter out the à. What am I doing wrong?

[mark]

You can't use \x notation. You have to use the actual chars.
<$noiselist = "a" "à"> works for me.
You need to enter the words for noise and equivs in whatever form they are coming from the browser. You could use
{query=<fmt "%U" $query>}
to get an idea what's coming in. Or save them to a file with
<write append /tmp/queries><fmt "%s\n" $query></write>

[edev]

Thanks Mark! I figured out the problem - our Java encoder was encoding the incoming utf-8 twice, so instead of putting "à" in the noiselist I have to put "Ã", then it works. Thank you for all your help!