wish to crawl pages for links but not include them in the index

jgdoke · Post by **jgdoke** » Tue May 13, 2008 12:22 pm

I need to crawl a site and have some pages which have the links to the real content be crawled for the links but not be included in the index.

For instance this page has the links I want in the index:
http://literature.rockwellautomation.co ... %20English

But I dont want that page in the index.
Hope that makes sense.
John

Post by **mark** » Tue May 13, 2008 12:32 pm

Use "Exclude by field". Create a metamorph query to match only those pages you want to affect. Use Exclude "Pages only" to exclude the page content but keep the links.

jgdoke · Post by **jgdoke** » Fri May 16, 2008 5:27 pm

I have searched for documentation on a metamorph query but did not find out how I would exclude pages with *browse* in the url. Can you help with that syntax?

Post by **mark** » Fri May 16, 2008 5:45 pm

Do you want to match the word or a substring (joebrowser)?
If the word enter
browse
If the substring enter
/browse

jgdoke · Post by **jgdoke** » Fri May 16, 2008 5:51 pm

there are two different url's tht I would like to match and only them.
http://literature.rockwellautomation.co ... /webassets /browse_results.hcst
http://literature.rockwellautomation.co ... egory.hcst

I need the links from them but not the page.

So if I enter browse_ that should catch both.

correct?

Post by **mark** » Mon May 19, 2008 10:42 am

Yes, and anything similar of course.