Thunderstone Support Forums

Hi,

We have been trialing the Webinator 4 software for a few days on a Win2k (fresh install) machine and are experiencing the software to hang randomly - mainly around 4500 (4000 - 4900) pages. The problem is not with any specific pages.

We would like to purchase the Professional Webinator Edition but would need to know that the software performs to our needs

Settings:

Base Url: http://www.dete.sa.gov.au/

Extensions:
.asp .html .htm .txt

Exclusions:

/sandpit/
/tester/
/odt/
/calendar/
/locs/

Strip Queries: N
Ignore Case: Y

Url Url:
http://203.152.225.167/webinator/dete.txt

Extra Domains (all on same server):
http://www.childcare.sa.gov.au
http://www.schools.sa.gov.au
http://www.training.sa.gov.au
http://www.employment.sa.gov.au
http://www.youth.sa.gov.au
http://www.schools.sa.gov.au/

Exclusion Prefix:
http://www.childcare.sa.gov.au/locs/
http://www.schools.sa.gov.au/locs/
http://www.training.sa.gov.au/locs/
http://www.employment.sa.gov.au/locs/
http://www.youth.sa.gov.au/locs/
http://www.schools.sa.gov.au/locs/

Index Name:

default.asp default.htm index.html index.htm

An urgent reply would be much appreciated as our CIO needs to make a decision soon.

Thanks in advance

I think something's getting lost in communication here. I just did a walk using the settings you specified and it went to successful completion and only found 597 good pages.

I have just run two more tests with only 1 Server and 1 instance - both of these searches hav hung on exactly the same page (and number of pages - 4431)

There is nothing wrong with the specific page itself

Please supply the base url and all of your non-default option settings so we can try to replicate the problem here.

I have uninstalled webinator and re-installed it.

After the re-install I ran the walk with exactly the settings above and it hung again after 5937 pages.

I can not complete a search with above settings without the walk to 'hang'. (i have also tried other settings on these domains with the same result)

Another problem which i have not mentioned before is that i can not stop the process after it hangs. Pressing the stop button does not stop the process - neither does a command line process.

Your webserver(s) are presenting themselves very differently to us than you so we can't really debug anything. We are able to do complete walks with your given settings with no problem, but the webservers present much fewer pages to us than to you. Why does your webserver provide a different view to the machine from which you are walking and other outside machines (like ours)? Is there some way you make it present the same view outside as inside so that we can have a chance of replicating your problem?

Hi Mark,

The reason why I re-installed the Webinator is that I can not understand why you are getting a differnet result.
The Machine that has Webinator on sit's on a completely different network.

The webservers are public and should present exactly the same informations to everybody.

R u running the walk from a *NIX or a Win32 box ? Could that make a difference - we unfortunately are bound to use Win32.

I've tried running the walk from unix and nt with the same results (success, no freeze). It shouldn't make a difference.
I've found that your webserver gives rather different results based on user-agent. Have you changed that?
Based on the info you've supplied I've guessed what you want to do and come up with the settings below. Create a new profile and try them.

Base URL
http://www.dete.sa.gov.au/
http://www.employment.sa.gov.au/employ/publish.asp
http://www.youth.sa.gov.au/youth/publish.asp
http://www.tafe.sa.gov.au/tafe/publish.asp
http://www.training.sa.gov.au/ovet_home.asp
http://www.childcare.sa.gov.au/cel_home.asp
http://www.schools.sa.gov.au/schl_home.asp

Extensions
.asp .html .htm .txt

Exclusions
/sandpit/
/tester/
/odt/
/calendar/
/locs/

Strip Queries
N

Ignore Case
Y

Extra Domains
www.dete.sa.gov.au
www.employment.sa.gov.au
www.youth.sa.gov.au
www.training.sa.gov.au
www.childcare.sa.gov.au
www.tafe.sa.gov.au
www.schools.sa.gov.au

Stay Under
N

Index Name
default.asp default.htm index.html index.htm

User Agent
Mozilla/4.51 [en] (X11; I; Linux 2.2.5-15 i686)

I just ran another walk with the settings above (using user agent: Mozilla/4.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) )

It hung again:

Basic Walk Settings
All Walk Settings
Search Settings
List/Edit URLs
Walk Status
Query Log
Test Search
Live Search
Profiles
Accounts
Documentation
Webinator Home
Logout
Free Webinator
Upgrade to Commercial Webinator

--------------------------------------------------------------------------------
Walk Status
Current Profile: test Webinator 4.0
Latest run: ABANDONED

Webinator Walk Report for test

Creating database e:\webinator/texis/test/db2...Done.
Walk started at 2001-11-22 07:03:38 (by user)
Start fetching at http://www.dete.sa.gov.au/
Start fetching at http://www.employment.sa.gov.au/employ/publish.asp
Start fetching at http://www.youth.sa.gov.au/youth/publish.asp
Start fetching at http://www.tafe.sa.gov.au/tafe/publish.asp
Start fetching at http://www.training.sa.gov.au/ovet_home.asp
Start fetching at http://www.childcare.sa.gov.au/cel_home.asp
Start fetching at http://www.schools.sa.gov.au/schl_home.asp
Ignore urls containing any of the following:
/sandpit/
/tester/
/odt/
/calendar/
/locs/
started 1 (1068) on http://www.dete.sa.gov.au/
started 2 (436) on http://www.employment.sa.gov.au/employ/publish.asp
30 pages fetched (166,745 bytes) from http://www.employment.sa.gov.au/employ/publish.asp
started 2 (304) on http://www.youth.sa.gov.au/
25 pages fetched (111,792 bytes) from http://www.youth.sa.gov.au/
started 2 (436) on http://www.tafe.sa.gov.au/tafe/publish.asp
24 pages fetched (76,948 bytes) from http://www.tafe.sa.gov.au/tafe/publish.asp
started 2 (820) on http://www.training.sa.gov.au/ovet_home.asp
4333 pages fetched (25,897,214 bytes) from http://www.dete.sa.gov.au/
started 2 (352) on http://www.employment.sa.gov.au/employ
2 pages fetched (1,432 bytes) from http://www.employment.sa.gov.au/employ
started 2 (328) on http://www.childcare.sa.gov.au/cel_home.asp
1685 pages fetched (11,644,620 bytes) from http://www.childcare.sa.gov.au/cel_home.asp
started 2 (352) on http://www.dete.sa.gov.au/
135 pages fetched (822,555 bytes) from http://www.dete.sa.gov.au/
started 2 (328) on http://www.schools.sa.gov.au/schlparents
113 pages fetched (776,985 bytes) from http://www.schools.sa.gov.au/schlparents
started 2 (352) on http://www.dete.sa.gov.au/
2 pages fetched (27,657 bytes) from http://www.dete.sa.gov.au/
started 2 (328) on http://www.tafe.sa.gov.au/tafe/
1 pages fetched (710 bytes) from http://www.tafe.sa.gov.au/tafe/
6509 pages (40,109,246 bytes).
744 errors.
512 duplicate pages.

6509 http://www.schools.sa.gov.au/schlparent ... navgrp=173 (550 bytes)
6508 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (98,611 bytes)
6507 http://www.schools.sa.gov.au/schlstuden ... sp?ext=cal (532 bytes)
6506 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (14,432 bytes)
6505 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (8,831 bytes)
6504 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (18,441 bytes)
6503 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (11,690 bytes)
6502 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (15,835 bytes)
6501 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (6,624 bytes)
6500 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (12,535 bytes)
6499 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (9,305 bytes)
6498 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (10,780 bytes)
6497 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (40,821 bytes)
6496 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (2,354 bytes)
6495 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (26,338 bytes)
6494 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (3,795 bytes)
6493 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (3,770 bytes)
6492 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (4,819 bytes)
6491 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (3,948 bytes)
6490 http://www.schools.sa.gov.au/schlstuden ... &navgrp=67 (3,934 bytes)
6489 http://www.schools.sa.gov.au/schlparent ... owback=173 (574 bytes)
6488 http://www.schools.sa.gov.au/schlparent ... navgrp=180 (17,925 bytes)
6487 http://www.schools.sa.gov.au/schlparent ... ?navgrp=65 (548 bytes)
6486 http://www.schools.sa.gov.au/schlparent ... navgrp=173 (5,082 bytes)
6485 http://www.schools.sa.gov.au/schlparent ... navgrp=173 (3,364 bytes)
6484 http://www.schools.sa.gov.au/schlparent ... navgrp=173 (5,211 bytes)
6483 http://www.schools.sa.gov.au/schlparent ... navgrp=173 (1,960 bytes)
6482 http://www.schools.sa.gov.au/schlparent ... navgrp=173 (5,987 bytes)
6481 http://www.schools.sa.gov.au/schlparent ... navgrp=173 (2,859 bytes)
6480 http://www.schools.sa.gov.au/schlstuden ... v_bar2.asp (11,414 bytes)
6479 http://www.schools.sa.gov.au/schlstuden ... ontent.asp (7,989 bytes)
6478 http://www.schools.sa.gov.au/schlparent ... howback=65 (574 bytes)
6477 http://www.schools.sa.gov.au/schlparent ... howback=65 (574 bytes)
6476 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (3,609 bytes)
6475 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (2,732 bytes)
6474 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (2,768 bytes)
6473 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (3,240 bytes)
6472 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (2,695 bytes)
6471 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (2,588 bytes)
6470 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (3,508 bytes)
6469 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (2,702 bytes)
6468 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (2,289 bytes)
6467 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (4,586 bytes)
6466 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (3,059 bytes)
6465 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (10,833 bytes)
6464 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (12,607 bytes)
6463 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (5,728 bytes)
6462 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (3,133 bytes)
6461 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (10,008 bytes)
6460 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (9,021 bytes)
6459 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (9,593 bytes)
6458 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (9,919 bytes)
6457 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (10,085 bytes)
6456 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (7,015 bytes)
6455 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (7,974 bytes)
6454 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (8,381 bytes)
6453 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (9,502 bytes)
6452 http://www.schools.sa.gov.au/schlparent ... &navgrp=65 (8,315 bytes)
6451 http://www.schools.sa.gov.au/schlparent ... v_bar2.asp (9,971 bytes)
6450 http://www.schools.sa.gov.au/schlstudents/ (539 bytes)
6449 http://www.schools.sa.gov.au/schlstaff/ (536 bytes)
6448 http://www.schools.sa.gov.au/schlparent ... ontent.asp (8,762 bytes)
6447 http://www.schools.sa.gov.au/schl_home.asp (5,436 bytes)
6446 http://www.schools.sa.gov.au/schl_contact/ (541 bytes)
6445 http://www.schools.sa.gov.au/schlparents (550 bytes)
6444 http://www.dete.sa.gov.au/offreview/fil ... ement.html (2,025 bytes)
6443 http://www.dete.sa.gov.au/lis/files/links/scheng.htm (3,042 bytes)
6442 http://www.dete.sa.gov.au/lis/files/links/invisweb.htm (2,941 bytes)
6441 http://www.dete.sa.gov.au/lis/files/links/eyconf.htm (2,850 bytes)
6440 http://www.dete.sa.gov.au/lis/files/links/ednsites.htm (18,184 bytes)
6439 http://www.dete.sa.gov.au/lis/files/links/confer.htm (7,175 bytes)
6438 http://www.dete.sa.gov.au/lis/files/links/ECassoc.htm (6,776 bytes)
6437 http://www.dete.sa.gov.au/lis/files/links/multiple.htm (14,989 bytes)
6436 http://www.dete.sa.gov.au/lis/files/links/lifelong.htm (6,031 bytes)
6435 http://www.dete.sa.gov.au/lis/files/links/emotion.htm (5,417 bytes)
6434 http://www.dete.sa.gov.au/lis/files/links/tlandtech.htm (8,374 bytes)
6433 http://www.dete.sa.gov.au/lis/files/links/learntech.htm (12,975 bytes)
6432 http://www.dete.sa.gov.au/lis/files/links/brain.htm (13,014 bytes)
6431 http://www.dete.sa.gov.au/lis/files/links/techlit.htm (7,584 bytes)
6430 http://www.dete.sa.gov.au/lis/files/links/speaklit.htm (8,047 bytes)
6429 http://www.dete.sa.gov.au/lis/files/lin ... venlit.htm (9,489 bytes)
6428 http://www.dete.sa.gov.au/lis/files/links/earlylit.htm (10,614 bytes)
6427 http://www.dete.sa.gov.au/lis/files/links/definlit.htm (4,002 bytes)
6426 http://www.dete.sa.gov.au/lis/files/links/critlit.htm (3,009 bytes)
6425 http://www.dete.sa.gov.au/lis/files/links/Equitylit.htm (7,195 bytes)
6424 http://www.dete.sa.gov.au/lis/files/links/Curriclit.htm (24,841 bytes)
6423 http://www.dete.sa.gov.au/lis/files/links/Assesslit.htm (9,929 bytes)
6422 http://www.dete.sa.gov.au/lis/files/links/techno.htm (2,869 bytes)
6421 http://www.dete.sa.gov.au/lis/files/links/society.htm (6,636 bytes)
6420 http://www.dete.sa.gov.au/lis/files/links/science.htm (13,857 bytes)
6419 http://www.dete.sa.gov.au/lis/files/links/numeracy.htm (12,630 bytes)
6418 http://www.dete.sa.gov.au/lis/files/links/maths.htm (22,279 bytes)
6417 http://www.dete.sa.gov.au/lis/files/links/lote.htm (3,367 bytes)
6416 http://www.dete.sa.gov.au/lis/files/links/infolit.htm (9,863 bytes)
6415 http://www.dete.sa.gov.au/lis/files/links/healthpe.htm (5,045 bytes)
6414 http://www.dete.sa.gov.au/lis/files/lin ... rprise.htm (4,360 bytes)
6413 http://www.dete.sa.gov.au/lis/files/links/english.htm (5,984 bytes)
6412 http://www.dete.sa.gov.au/lis/files/links/currint.htm (5,203 bytes)
6411 http://www.dete.sa.gov.au/lis/files/links/curdev.htm (5,673 bytes)
6410 http://www.dete.sa.gov.au/lis/files/links/civics.htm (14,288 bytes)
--------------------------------------------------------------------------------
Previous finished run:
-None-




I will do another walk using User Agent Mozilla/4.51 [en] (X11; I; Linux 2.2.5-15 i686) and compare the result.

Very odd. I only get about 1000 pages total...
When that completes, could you upload your database to
ftp://ftp.thunderstone.com/pub/people/waynesdb/
so we can compare what it found vs what we found.