On one of our servers, we have a robots.txt file that has the following lines:
User-agent: *
Disallow: /
When Webinator 4.1 parses the robots.txt file I get the following output
The following exclusion(s) specified by robots.txt will be ignored as they would exclude the base url (http://ntsrvr1.mysite.com/).
/
The problem is we want everything to be excluded from this site. This is why we made a Disallow: / robots.txt file. What can I do to stop Webinator from indexing sites that have their robots.txt file like the one I have included. I don't want phone calls and emails from site administrators upset because their site is being walked.
User-agent: *
Disallow: /
When Webinator 4.1 parses the robots.txt file I get the following output
The following exclusion(s) specified by robots.txt will be ignored as they would exclude the base url (http://ntsrvr1.mysite.com/).
/
The problem is we want everything to be excluded from this site. This is why we made a Disallow: / robots.txt file. What can I do to stop Webinator from indexing sites that have their robots.txt file like the one I have included. I don't want phone calls and emails from site administrators upset because their site is being walked.