Ignoring robots.txt file

sunnedaze · Post by **sunnedaze** » Thu Jul 31, 2003 11:24 am

I guess I don't know how to do that test. I typed it at the RUN prompt (I am on NT) and there was a too quick to catch display.

texis "profile=allenet" top="cleohsenet01.napa.adpa.ad.etn.com" ttyverbose=1 dowalk/getrobots.txt

Post by **mark** » Thu Jul 31, 2003 12:28 pm

And an http url requires the leading "http://"

sunnedaze · Post by **sunnedaze** » Thu Jul 31, 2003 12:50 pm

texis "profile=allenet" top="cleohsenet01.napa.adpa.ad.etn.com" ttyverbose=1 dowalk/getrobots.txt
texis is not recognized as an internal or external command, operable program or batch file

Post by **mark** » Thu Jul 31, 2003 1:09 pm

http://www.thunderstone.com/texis/site/ ... ing+dowalk

sunnedaze · Post by **sunnedaze** » Thu Jul 31, 2003 2:30 pm

Sorry to be such a dummy. We have the dowalk script installed at F:/Thunderstone Software/Webinator/Texis/Scripts/Webinator.

We have the PROFILENAME installed at F:/Thunderstone Software/Webinator/Texis

Doing a search on the Document Root (URL of website), I'm not coming up with anything.

Tried running texis profile=PROFILENAME" dowalk/dispatch.txt at both prompts & same message....texis is not recognized, etc.

Post by **mark** » Thu Jul 31, 2003 3:05 pm

F:\Thunderstone Software\Webinator\Texis is not the profile name. The profile name is the name you gave your webinator walk profile when you created it in the admin interface.

You should find texis.exe in your installation directory.

F:\Thunderstone Software\Webinator\Texis\texis profile=allenet top=http://cleohsenet01.napa.adpa.ad.etn.com "F:/Thunderstone Software/Webinator/Texis/Scripts/Webinator/dowalk/getrobots.txt"

or to shorten it up cd to the install directory first

cd F:\Thunderstone Software\Webinator\Texis
texis profile=allenet top=http://cleohsenet01.napa.adpa.ad.etn.com Scripts/Webinator/dowalk/getrobots.txt

sunnedaze · Post by **sunnedaze** » Thu Jul 31, 2003 3:22 pm

I got a listing of a bunch of texis options & then back to the prompt. Viewing the walk log via the browser shows nothing was updated.

Post by **mark** » Thu Jul 31, 2003 6:16 pm

Nevermind, it appears there was a potential problem parsing robots.txt on windows. Download the latest dowalk and webinatoradmin from the webinator examples page for a fix.

sunnedaze · Post by **sunnedaze** » Fri Aug 01, 2003 11:13 am

Installed the 2 scripts. Ran a new walk. Still no robots.txt file in the log. (I did not reboot after the install.)
Creating database F:\Thunderstone Software\Webinator/texis/allenet/db2...Done.
Walk started at 2003-08-01 11:09:24 (by user)
JavaScript walking not enabled by current license
HTTPS walking disabled
Start fetching at http://cleohsenet01.napa.ad.etn.com/
Reading urls from file F:\iPlanet\Servers\docs\enet\html\eneturl.txt
Ignore urls containing any of the following:
/cgi-bin/
~
?

started 1 (3744) on http://cleohsenet01.napa.ad.etn.com/
6118 pages (135,234,774 bytes) so far.
223 errors so far.
24 duplicate pages so far.

sunnedaze · Post by **sunnedaze** » Fri Aug 01, 2003 11:17 am

Could be the file is named wrong? The new script files are .txt. The original ones are properties=file.