can't rewalk

djeckel · Post by **djeckel** » Mon Apr 30, 2001 2:45 pm

I just checked w/ our sys admin... we have three logs: system, security, application; none of which record anything when I try and rewalk.

Post by **mark** » Mon Apr 30, 2001 4:07 pm

None of those sound like webserver logs. I'm not your web site admin so I can't tell you where the logs are exactly. You might look under something like c:\winnt\system32\logfiles or check the webserver's configuration for where it keeps it's logs.

djeckel · Post by **djeckel** » Mon Apr 30, 2001 4:32 pm

There's nothing in the server logs. Is webinator actually making a request?

Post by **mark** » Mon Apr 30, 2001 6:00 pm

That's the only way it could have gotten that message. Let's start clean so we know what's going on. Forget -rewalk for the moment and try to simply walk the url in question to a new scratch database.

gw -djunk -wipe
gw -djunk -v9 -p5 http://www.slackinc.com/

If you still get the "forbidden" or similar message check the webserver logs and configuration on host www.slackinc.com to see why.

djeckel · Post by **djeckel** » Tue May 01, 2001 4:04 pm

ok... after "gw -djunk -wipe " it returned:
Disallowed protocol
Visited 0 pages total

after "gw -djunk -v9 -p5 http://www.slackinc.com/ " it returned a whole lot of stuff..
created "junk" in the "bin" directory and came back with the same 403 error

I still can't find anything in the server logs. It all looks like normal traffic. The only mention of webinator is people trying to use it.

djeckel · Post by **djeckel** » Tue May 01, 2001 4:11 pm

I was wrong... I found 2 lines

2001-05-01 18:59:59 212.253.6.246 - 206.1.96.2 POST /scripts/texis.exe - 200 1911 620 111 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+5.0;+Windows+98;+DigExt) - http://www.slackinc.com/scripts/texis.e ... r/search2/

2001-05-01 19:00:39 212.253.6.246 - 206.1.96.2 POST /scripts/texis.exe - 200 1911 620 130 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+5.0;+Windows+98;+DigExt) - http://www.slackinc.com/scripts/texis.e ... r/search2/

Post by **John** » Tue May 01, 2001 4:50 pm

Those two lines seem to be normal searches. Note also that since most of your pages are .asp pages you will need to add -fasp to your command line. Is the homepage an ASP that does any permission checks?

Post by **mark** » Tue May 01, 2001 5:16 pm

Please send the entirety of the "whole lot of stuff" to help@thunderstone.com for analysis. Also include the complete output of "gw -version".

Check your webserver log for all accesses from the machine from which you are running gw to see what it's returning to gw.

BTW, you must have typed something other than "gw -djunk -wipe" to get the "Disallowed protocol" message.

djeckel · Post by **djeckel** » Wed May 02, 2001 2:17 pm

I apologize.. I guess I did type it wrong.
After "gw -djunk -wipe" I got:
Wiping database

Here's "gw -version"
Webinator WWW Site Indexer Version 2.56 (Commercial)
Copyright(c) 1995,1996,1997,1998,1999,2000 Thunderstone EPI Inc.
Release: 20001030
Use: gw [options] URL
or : gw -s[X] [options] SQL
-h : help.
url : Where to begin fetching HTML from.

There are no permission checks on the homepage.

djeckel · Post by **djeckel** » Thu May 03, 2001 2:16 pm

I tried "gw -create -fasp -v9" and got this. suggestions? thanks.

Option: "-create"
Option: "-fasp"
Option: "-v9"
No database specified. Use the default (d:/wwwroot/webinator/db)?
(y/n) default is y : You may use "-d-" to skip this question in the future.
100 Database d:/wwwroot/webinator/db already exists
200 Using existing database
200 Using existing SYSOBJECTS
create table html(id counter,New int,Visited date,Dlsecs int,Depth int,Url varchar(80),Title varchar(80),Body blob,Meta blob);
176 Table html already exists in data dictionary
000 SQLPrepare() failed with -1
create table refs(id counter,Url varchar(80),Ref varchar(80));
176 Table refs already exists in data dictionary
000 SQLPrepare() failed with -1
create table todo(Dlsecs int,Depth int,Url varchar(80));
176 Table todo already exists in data dictionary
000 SQLPrepare() failed with -1
create table error(id counter,Url varchar(80),Reason varchar(80));
176 Table error already exists in data dictionary
000 SQLPrepare() failed with -1
create table options(id counter,Profile varchar(8),Name varchar(8),Type varchar(8),Int int,Float double,String varchar(80),Strlist strlst);
176 Table options already exists in data dictionary
000 SQLPrepare() failed with -1
create table querylog(id counter,Client varchar(40),Query varchar(80));
176 Table querylog already exists in data dictionary
000 SQLPrepare() failed with -1
create unique index xhtmlurl on html(Url);
100 BTREE Index d:\wwwroot\webinator\db\xhtmlurl already exists on html(Url)
000 SQLPrepare() failed with -1
getip() called 0 times. 0 hits
gethostbyname() called 0 times