crawling 2 sites - 1 profile or 2?

john.santangelo · Post by **john.santangelo** » Tue Jan 22, 2008 9:08 am

My question is how to most efficiently set up crawling of 3 web sites.

We have profile 1 = our site 1 + external site
we also have profile 2 = our site 2 + the same external site. Is that the best setup (to crawl the same external site twice) or can we set up the external site as a third profile, crawl once, and somehow copy the data to the existing 2 profiles?

Post by **mark** » Tue Jan 22, 2008 10:45 am

You could crawl each separately (3 profiles) and setup 2 more meta profiles to search and combine results from the desired of the first 3 profiles.

john.santangelo · Post by **john.santangelo** » Tue Jan 22, 2008 11:31 am

How long does it take from the setup of the meta profile to search the existing regular profiles?

I'm getting "No documents match the query." for all searches, but no errors.

Also for Host IP or name can I use a DNS alias instead of IP?

Post by **mark** » Tue Jan 22, 2008 12:08 pm

No time. If the regular profile searches are working a meta search of those profiles should work immediately. Make sure you've set the "visible" flag under search settings for the regular profiles though.

You can use any IP or name that the appliance can resolve to the proper machine. But in the case of a single appliance meta searching itself you'd generally use "localhost" or "127.0.0.1".

john.santangelo · Post by **john.santangelo** » Tue Jan 22, 2008 1:07 pm

Perfect. Its working now. I missed the "visible" step.
One last question. How to set categories in a meta search profile?

Post by **mark** » Tue Jan 22, 2008 2:39 pm

Sorry, meta searches don't currently support categories.

john.santangelo · Post by **john.santangelo** » Tue Jan 22, 2008 2:47 pm

well then i'm back to crawling the external site twice then.