webinator hanging randomly

waynes · Post by **waynes** » Mon Nov 19, 2001 5:02 pm

Hi,

We have been trialing the Webinator 4 software for a few days on a Win2k (fresh install) machine and are experiencing the software to hang randomly - mainly around 4500 (4000 - 4900) pages. The problem is not with any specific pages.

We would like to purchase the Professional Webinator Edition but would need to know that the software performs to our needs

Settings:

Base Url: http://www.dete.sa.gov.au/

Extensions:
.asp .html .htm .txt

Exclusions:

/sandpit/
/tester/
/odt/
/calendar/
/locs/

Strip Queries: N
Ignore Case: Y

Url Url:
http://203.152.225.167/webinator/dete.txt

Extra Domains (all on same server):
http://www.childcare.sa.gov.au
http://www.schools.sa.gov.au
http://www.training.sa.gov.au
http://www.employment.sa.gov.au
http://www.youth.sa.gov.au
http://www.schools.sa.gov.au/

Exclusion Prefix:
http://www.childcare.sa.gov.au/locs/
http://www.schools.sa.gov.au/locs/
http://www.training.sa.gov.au/locs/
http://www.employment.sa.gov.au/locs/
http://www.youth.sa.gov.au/locs/
http://www.schools.sa.gov.au/locs/

Index Name:

default.asp default.htm index.html index.htm

An urgent reply would be much appreciated as our CIO needs to make a decision soon.

Thanks in advance

Post by **mark** » Mon Nov 19, 2001 6:22 pm

I think something's getting lost in communication here. I just did a walk using the settings you specified and it went to successful completion and only found 597 good pages.

waynes · Post by **waynes** » Mon Nov 19, 2001 6:27 pm

I have just run two more tests with only 1 Server and 1 instance - both of these searches hav hung on exactly the same page (and number of pages - 4431)

There is nothing wrong with the specific page itself

Post by **mark** » Mon Nov 19, 2001 11:06 pm

Please supply the base url and all of your non-default option settings so we can try to replicate the problem here.

waynes · Post by **waynes** » Tue Nov 20, 2001 1:27 am

I have uninstalled webinator and re-installed it.

After the re-install I ran the walk with exactly the settings above and it hung again after 5937 pages.

I can not complete a search with above settings without the walk to 'hang'. (i have also tried other settings on these domains with the same result)

Another problem which i have not mentioned before is that i can not stop the process after it hangs. Pressing the stop button does not stop the process - neither does a command line process.

Post by **mark** » Tue Nov 20, 2001 10:01 am

Your webserver(s) are presenting themselves very differently to us than you so we can't really debug anything. We are able to do complete walks with your given settings with no problem, but the webservers present much fewer pages to us than to you. Why does your webserver provide a different view to the machine from which you are walking and other outside machines (like ours)? Is there some way you make it present the same view outside as inside so that we can have a chance of replicating your problem?

waynes · Post by **waynes** » Tue Nov 20, 2001 10:22 pm

Hi Mark,

The reason why I re-installed the Webinator is that I can not understand why you are getting a differnet result.
The Machine that has Webinator on sit's on a completely different network.

The webservers are public and should present exactly the same informations to everybody.

R u running the walk from a *NIX or a Win32 box ? Could that make a difference - we unfortunately are bound to use Win32.

Post by **mark** » Wed Nov 21, 2001 12:44 pm

I've tried running the walk from unix and nt with the same results (success, no freeze). It shouldn't make a difference.
I've found that your webserver gives rather different results based on user-agent. Have you changed that?
Based on the info you've supplied I've guessed what you want to do and come up with the settings below. Create a new profile and try them.

Base URL
http://www.dete.sa.gov.au/
http://www.employment.sa.gov.au/employ/publish.asp
http://www.youth.sa.gov.au/youth/publish.asp
http://www.tafe.sa.gov.au/tafe/publish.asp
http://www.training.sa.gov.au/ovet_home.asp
http://www.childcare.sa.gov.au/cel_home.asp
http://www.schools.sa.gov.au/schl_home.asp

Extensions
.asp .html .htm .txt

Exclusions
/sandpit/
/tester/
/odt/
/calendar/
/locs/

Strip Queries
N

Ignore Case
Y

Extra Domains
www.dete.sa.gov.au
www.employment.sa.gov.au
www.youth.sa.gov.au
www.training.sa.gov.au
www.childcare.sa.gov.au
www.tafe.sa.gov.au
www.schools.sa.gov.au

Stay Under
N

Index Name
default.asp default.htm index.html index.htm

User Agent
Mozilla/4.51 [en] (X11; I; Linux 2.2.5-15 i686)

waynes · Post by **waynes** » Wed Nov 21, 2001 5:30 pm

Post by **mark** » Wed Nov 21, 2001 5:52 pm

Very odd. I only get about 1000 pages total...
When that completes, could you upload your database to
ftp://ftp.thunderstone.com/pub/people/waynesdb/
so we can compare what it found vs what we found.