In all of their "wisdom", Geocities has within the last month decided to block
all robots from their entire site except for certain "well-behaved" robots.
I really do not want to disrespect the robots.txt files on all the sites I
search just because of Geocities, so would it be possible for you to either:
(1) Make it possible to allow the "-r" command to apply for a specific site
(like the "-j" command does), or
(2) Try to get Webinator included in Geocities "list of allowed robots" that
is in their robots.txt file.
Or, if you know of any other solutions other than using the "-r" command, I
would appreciate it.
Winnipeg, Manitoba, Canada
Web site: http://www.concentric.net/~lkessler