Selective Robot Exclusion

Post Reply
User avatar
Thunderstone
Site Admin
Posts: 2504
Joined: Wed Jun 07, 2000 6:20 pm

Selective Robot Exclusion

Post by Thunderstone »




In all of their "wisdom", Geocities has within the last month decided to block
all robots from their entire site except for certain "well-behaved" robots.
See:
http://www.geocities.com/robots.txt

I really do not want to disrespect the robots.txt files on all the sites I
search just because of Geocities, so would it be possible for you to either:

(1) Make it possible to allow the "-r" command to apply for a specific site
(like the "-j" command does), or

(2) Try to get Webinator included in Geocities "list of allowed robots" that
is in their robots.txt file.

Or, if you know of any other solutions other than using the "-r" command, I
would appreciate it.

Louis Kessler
Winnipeg, Manitoba, Canada

E-mail: lkessler@concentric.net
Web site: http://www.concentric.net/~lkessler



User avatar
Thunderstone
Site Admin
Posts: 2504
Joined: Wed Jun 07, 2000 6:20 pm

Selective Robot Exclusion

Post by Thunderstone »




-r applies to all hosts walked within a run. You can walk the desired
site in a separate run (different options, same database) to get
different behavior for that site.


Post Reply