misc &url.lst notes

Post Reply
User avatar
Thunderstone
Site Admin
Posts: 2504
Joined: Wed Jun 07, 2000 6:20 pm

misc &url.lst notes

Post by Thunderstone »



Greetings,

Is it just me, or is there no mention of &url.lst in the Webinator 2.1
admin documentation? I just stumbled accross it in the old Webinator 1.3
admin doc (to which a link still exists off the Tech Support Archive
page). This is great, because it's exactly what I need.

Does Webinator 2.1 fix the old problem of not being able to deal with
multiple robots.txt files for different URLs? I'd prefer to deal with
sites by the rules that they set as opposed to slamming them with multiple
hits.

Best 'n all,

Mark Weixel

| Mark J. Weixel | "To new shores! Despite the tempest, the |
| CIS Support Services | shoals and the hidden rocks... |
| 110 Old Engineering Hall | To new shores!" |
| weixel+@pitt.edu | --Musorgskij |



Post Reply