I've noted that Webinator will follow the rules
defined in the robots.txt file. But what about
the robot meta tag stuff? For example:
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW">
Does webinator eval these directives?
-thanks
defined in the robots.txt file. But what about
the robot meta tag stuff? For example:
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW">
Does webinator eval these directives?
-thanks