robots.txt allow statements

Post Reply
mjacobson
Posts: 204
Joined: Fri Feb 08, 2002 3:35 pm

robots.txt allow statements

Post by mjacobson »

Does Webinator follow "Allow" statements in a site's robots.txt file?
Top
User avatar
mark
Site Admin
Posts: 5519
Joined: Tue Apr 25, 2000 6:56 pm

robots.txt allow statements

Post by mark »

No. I don't think that ever got to be a standard?
Top
mjacobson
Posts: 204
Joined: Fri Feb 08, 2002 3:35 pm

robots.txt allow statements

Post by mjacobson »

No it is not part of the standard. I was just wondering if you guys had done anything with it. I am seeing more robots.txt files having Allow statements so I thought I would ask.
Top
User avatar
John
Site Admin
Posts: 2613
Joined: Mon Apr 24, 2000 3:18 pm
Location: Cleveland, OH
Contact:
Contact John

Re: robots.txt allow statements

Post by John »

Note that Allow was added to the robots.txt rule processing in Version 14
John Turnbull
Thunderstone Software
Top
Post Reply

Return to “Webinator”