Indexing question

Post by **Thunderstone** » Thu Apr 15, 1999 12:19 pm

I have just installed the latest free version of webinator on a Sun solaris
7 system.

When I try to index a subtree of my website with:

gw http://computer.x.edu/subdir/index.html

or just ../subdir

It consistently indexes directories above this level, even if they are not
referenced in pages from the subdir. I even have a robots.txt in one
of them and it still indexes that directory.

How can I index just a subtree?

Thanks,John Rutherford

Post by **Thunderstone** » Thu Apr 15, 1999 12:49 pm

You need to use the -j option.
gw -jhttp://computer.x.edu/subdir/ http://computer.x.edu/subdir/index.html
See http://www.thunderstone.com/gw25man/node58.html

gw will not walk anything that is not referenced, so there must be
at least one reference out of the desired directory if it's wandering out.

robots.txt only goes in the document root directory and may contain
specifications for any directory on the site. robots.txt in a subdirectory
has no meaning at all.