Walk does not index anymore

michel.weber · Post by **michel.weber** » Wed Jun 20, 2007 6:20 pm

Hi

Our main site is not indexed anymore : http://www.coe.int
In the walk status file it says 'cannot connect to http://www.coe.int'. Any idea?

The site seems to be perfectly accessible

-------------------------------------------------------

Search Appliance Walk Report for y_www

Creating database /usr/local/morph3/texis/ywww.4679af99334d/db2...Done.
Walk started at 2007-06-21 00:16:09 (by user)
JavaScript walking enabled
HTTPS walking enabled
Start fetching at http://www.coe.int/
Ignore urls containing any of the following:
~
/_vti
/cgi-bin/
/Wires/
dev.
test
ToPrint=yes
WcdDoc.asp
WCDsearch.asp
asp?link=http
2007-06-21 00:16:09 started 1 new (20104) on http://www.coe.int/

006 /usr/local/morph3/texis/scripts/dowalk(doprimer) 400: Cannot connect to www.wysistat.com:80: Connection refused
0 pages fetched (87,024 bytes) from http://www.coe.int/
1 errors
0 duplicate pages
No pages fetched. Search not updated.

The link : http://www.coe.int/
Had this error: Cannot connect to www.wysistat.com:80: Connection refused

Post by **mark** » Thu Jun 21, 2007 10:02 am

It's complaining about the server containing javascript for that page, link "http://www.wysistat.com/statistique.js". Apparently www.wysistat.com was offline when you did the walk. It appears to be online now. Try again.

michel.weber · Post by **michel.weber** » Fri Jun 22, 2007 12:52 pm

Well yessss, but the profile is configured so as not to fetch offsite pages, so it should not follow that link???

Post by **mark** » Fri Jun 22, 2007 1:22 pm

That's not considered a link (reference to a different page), but a fundamental part of the page being processed. Most people want these portions included even if they don't want offsite pages.

If javascript is not required to navigate the site or include important content you could turn off javascript processing.

michel.weber · Post by **michel.weber** » Mon Jun 25, 2007 12:56 pm

Unfortunately i can not just switch off javascript as navigation is based on it.
the http://www.wysistat.com/statistique.js refers to an external script for statistics qhich unfortunately does not work with indexers.

Is there a way to instruct thunderstone to completely ignore anything between 2 text strings? I tried to use 'ignore tags', but that does not seem to work.

Post by **mark** » Tue Jun 26, 2007 1:50 pm

Ignore tags removes text but not links (it's usually used for menus etc). I can't think of any way of preventing that javascript except for turning off script fetching. Are you sure you need that? I'm able to navigate that site with javascript turned off in the browser.

michel.weber · Post by **michel.weber** » Tue Jun 26, 2007 3:03 pm

Unfortunately the menu system is at least partially based on javascript.

I'll see with the webmaster what can be done.

Is there a chance that to get 'offsite' javascripts treaed as other 'offsite' pages in a future release?

Post by **mark** » Tue Jun 26, 2007 6:05 pm

Probably not in the near term.