Thunderstone Support Forums

An external site is complaining to us about 1000s of hits Webinator seems to generate every day.

I have tried many things to stop this from happening as we don't want to index this site anyway.
These are the things I have done so far (in this order):
1. Removed the url from Base URL (under all walks)
2. Made sure robots.txt was set to Y
3. Deleted the whole profile and re-created it again (without this external url and the above settings) and reindexed.

Besides only two rather drastic measures (below) I am at a loss what to do:
A. asking this external company to stop all requests coming from the search engine's site (IP address) with agent string Mozilla/4.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)";
B. remove Webinator and carry out a complete reinstall.

PS. This problem started when I installed the latest version of Webinator (V5.1 with PDF/Word plugin).

Thanks a lot,

Jan

What other non-default settings are you using? By default webinator will stay on the sites mentioned in base url. If some of your settings are allowing the other site you could add that site to the excludes list.

Have you tried downloading the latest scripts from the website to see if that changes anything?

On top of the above settings I have done the following:
- put the external url in the exclusions list (without
the http://www bit).
- Max depth set to 2
- Off site pages to N

The scripts I'm using aren't the most up to date ones as they are heavily customised. So, new scripts would again need a lot of time spending on them.
However the search is customised a lot, but the dowalk ones isn't that much. Would I be able to use a new dowalk script without changing the search script or is this not recommended?
I assume here that dowalk is the one causing the problems.

Thanks again!

Upgrading dowalk (within the same major version number) shouldn't be a problem for search. Don't use the index fields feature if your search is based on something older than 5.1.5.

That sounds normal and should work as expected unless the dowalk script has been changed. If you provide actual values I could try replicating it here.

Dowalk has now been upgraded to its latest version.
I have also created a copy of the oiginal profile, indexed this and removed the original one and am still experiencing the many hits on this external site from Webinator.
Replicating the profile on a different box doesn't actually increase the hits.

Bar carrying out a complete re-install I'm at a complete loss what to do.

Any advice?

Thanks

Maybe the site in question goes by more than one name and you're only blocking one of them but allowing the others?

If you provide actual values we might have a better idea or could try replicating it here. Compare your profile settings to an entirely default profile and let us know everything that's different than defaults.

Silly question... These webinator hits are coming from your machine right? Not someone else also using Webinator?

Thanks Mark, I am assuming the Webinator requests are coming from the machine we're thinking of but now I'm not too sure anymore.
Would it be possible to disable Webinator for an hour or so to check if it is actually the right machine we're looking at?

So far I've tried monitor.exe and texis.exe but haven't managed to disabled it yet. Possibly renaming a exe for a little while may be?

This seems much easier than having to reinstall it and then finding out that it actually didn't do the trick.

I'm more tempted to think that this external site has its own wires crossed a bit as we've enabled a network sniffer to track the network traffic between us and them. The result was that there is actually minimal network traffic!
Thanks!

Simplest is to unschedule any scheduled profiles then do "pause and live" for any profiles that are currently running.

If you want to ensure that nothing can be restarted rename texis.exe in both the INSTALLDIR and the CGI directory.