I would like to exclude all IP address from being crawled. I am using gw and in the options file I added the following regexp x/>>=\digit{3}\. Is there a flag or some other way to keep webinator from walking any links that are IP address based?
Don't walk IP address
-
- Site Admin
- Posts: 5519
- Joined: Tue Apr 25, 2000 6:56 pm
Don't walk IP address
There's no option or such designed for doing that. -x/ only applies to the path, not the hostname (or IP) part of the url. You'd have to use a walk script or some sort.