Page 1 of 1

robots.txt allow statements

Posted: Wed Dec 08, 2004 12:45 pm
by mjacobson
Does Webinator follow "Allow" statements in a site's robots.txt file?

robots.txt allow statements

Posted: Wed Dec 08, 2004 2:10 pm
by mark
No. I don't think that ever got to be a standard?

robots.txt allow statements

Posted: Wed Dec 08, 2004 2:14 pm
by mjacobson
No it is not part of the standard. I was just wondering if you guys had done anything with it. I am seeing more robots.txt files having Allow statements so I thought I would ask.

Re: robots.txt allow statements

Posted: Fri Mar 05, 2021 2:15 pm
by John
Note that Allow was added to the robots.txt rule processing in Version 14