Page 1 of 1
robots.txt allow statements
Posted: Wed Dec 08, 2004 12:45 pm
by mjacobson
Does Webinator follow "Allow" statements in a site's robots.txt file?
robots.txt allow statements
Posted: Wed Dec 08, 2004 2:10 pm
by mark
No. I don't think that ever got to be a standard?
robots.txt allow statements
Posted: Wed Dec 08, 2004 2:14 pm
by mjacobson
No it is not part of the standard. I was just wondering if you guys had done anything with it. I am seeing more robots.txt files having Allow statements so I thought I would ask.
Re: robots.txt allow statements
Posted: Fri Mar 05, 2021 2:15 pm
by John
Note that Allow was added to the robots.txt rule processing in
Version 14