Page 1 of 1

Include cgi-bin URLs

Posted: Wed Jan 03, 2007 1:57 pm
by Phill-26
I'd like to include URLs that contain cgi-bin and as this is a default exlcusion I am removing it from the exclusion box in walk settings but the walk status page is still saying it will ignore cgi-bin URLs.
I am only using single page walks and have noticed it is walking some cgi-bin URLs but missing some others.

Any thoughts appreciated.

Include cgi-bin URLs

Posted: Wed Jan 03, 2007 2:13 pm
by mark
You might want to do a mode "new" walk to get a clean slate instead of "refresh".
Also make sure robots.txt on that site doesn't exclude cgi-bin. If it does you can turn off robots.txt in the walk settings.