This page: http://www.rockwellautomation.com/process/ has a robots meta tag set to noindex. I have set the crawl to obey Meta Robots:
Robots robots.txt: Y Meta: Y Placeholder: Y
This is the entry in List url's:
URL: http://www.rockwellautomation.com/process/
Depth: 1 click away from Home
Size: 456 bytes download, 0 bytes text
Indexed: 2008-07-17 01:04:52
Modified: 2008-07-17 01:04:52
Last Visit: 2008-07-17 01:04:52
Next Visit: 2008-07-24 01:04:52 Update Soon
Download time: Less than 1 second
Hash: 487ee1046f
Links: Parents, Children
Error: meta robots NOINDEX
Categories: -None-
Title: -None-
Description: -None-
Keywords: -None-
Meta data: -None-
Text Charset: -None-
Source MIME Type: -None-
But in Searches this page still shows up:
http://search2.rockwellautomation.com/t ... ry=process
Appliance scripts versions
Version: Search Appliance Server Version 5.01.1196094852 20071126 (i686-unknown-linux2.4.9-64-32)
Scripts Version: 6.3.1
Ideas?
Robots robots.txt: Y Meta: Y Placeholder: Y
This is the entry in List url's:
URL: http://www.rockwellautomation.com/process/
Depth: 1 click away from Home
Size: 456 bytes download, 0 bytes text
Indexed: 2008-07-17 01:04:52
Modified: 2008-07-17 01:04:52
Last Visit: 2008-07-17 01:04:52
Next Visit: 2008-07-24 01:04:52 Update Soon
Download time: Less than 1 second
Hash: 487ee1046f
Links: Parents, Children
Error: meta robots NOINDEX
Categories: -None-
Title: -None-
Description: -None-
Keywords: -None-
Meta data: -None-
Text Charset: -None-
Source MIME Type: -None-
But in Searches this page still shows up:
http://search2.rockwellautomation.com/t ... ry=process
Appliance scripts versions
Version: Search Appliance Server Version 5.01.1196094852 20071126 (i686-unknown-linux2.4.9-64-32)
Scripts Version: 6.3.1
Ideas?