Does Webinator follow "Allow" statements in a site's robots.txt file?
robots.txt allow statements
-
mark
- Site Admin
- Posts: 5519
- Joined: Tue Apr 25, 2000 6:56 pm
robots.txt allow statements
No. I don't think that ever got to be a standard?
-
mjacobson
- Posts: 204
- Joined: Fri Feb 08, 2002 3:35 pm
robots.txt allow statements
No it is not part of the standard. I was just wondering if you guys had done anything with it. I am seeing more robots.txt files having Allow statements so I thought I would ask.
-
John
- Site Admin
- Posts: 2623
- Joined: Mon Apr 24, 2000 3:18 pm
- Location: Cleveland, OH
Re: robots.txt allow statements
Note that Allow was added to the robots.txt rule processing in Version 14
John Turnbull
Thunderstone Software
Thunderstone Software