Exclude index pages

pete.smith · Post by **pete.smith** » Wed Jun 28, 2006 12:42 pm

Hi here is my issue - I have a frontend for newsgroups that I am crawling. I really only want THE RESULTS (the posts) not the indeces of posts (the thread view) in my results. Urls are pretty standard like thread.php?= or ?article.php? I just want articles.

Post by **mark** » Wed Jun 28, 2006 12:55 pm

Ideally the index pages would have
<meta name="robots" content="noindex,follow">
on them. See http://www.robotstxt.org/wc/meta-user.html

If you can't control that you can use "Data from Field" to simulate it. Set it to recognize your index page urls or content and set "Exclude" to "Pages only".

Post by **John** » Wed Jun 28, 2006 1:52 pm

With exclude by field you could search for thread.php in the URL and exclude pages only, not links.

Post by **mark** » Wed Jun 28, 2006 1:54 pm

oops, I meant "Exclude by field" not "Data from field".