Page 1 of 1
Indexing question
Posted: Thu Apr 15, 1999 12:19 pm
by Thunderstone
I have just installed the latest free version of webinator on a Sun solaris
7 system.
When I try to index a subtree of my website with:
gw
http://computer.x.edu/subdir/index.html
or just ../subdir
It consistently indexes directories above this level, even if they are not
referenced in pages from the subdir. I even have a robots.txt in one
of them and it still indexes that directory.
How can I index just a subtree?
Thanks,John Rutherford
Indexing question
Posted: Thu Apr 15, 1999 12:49 pm
by Thunderstone
You need to use the -j option.
gw -jhttp://computer.x.edu/subdir/
http://computer.x.edu/subdir/index.html
See
http://www.thunderstone.com/gw25man/node58.html
gw will not walk anything that is not referenced, so there must be
at least one reference out of the desired directory if it's wandering out.
robots.txt only goes in the document root directory and may contain
specifications for any directory on the site. robots.txt in a subdirectory
has no meaning at all.