fine tuning performance.

Post Reply
agarcian
Posts: 15
Joined: Wed Oct 20, 2004 2:07 pm

fine tuning performance.

Post by agarcian »

I have a few main subfolders that I want to index and I need to detect the changes as often as I can, let's say each hour. Since each subfolder can contain more than 100 thousand documents, I am guessing that if I split each subfolder in different profiles I can gain a better performance. Can this be true?

If so, would it be possible to generate a single query across profiles? Does it make sense to separate the crawling into different profiles seeking performance and run a query accross profiles?

Thanks.
User avatar
Kai
Site Admin
Posts: 1271
Joined: Tue Apr 25, 2000 1:27 pm

fine tuning performance.

Post by Kai »

Assuming you are crawling file:// URLs, you might gain some rewalk performance splitting across profiles, when some of the profiles have few or no changed documents (because then an index rebuild is faster or doesn't need to happen). But all the documents still need to be downloaded to determine if they have changed, which takes time. A query running across multiple profiles would probably not be any faster than a single query against a merged profile.

We are working on an upgrade to support If-Modified-Since crawling with file:// URLs, so that the entire document does not need to be downloaded if it has not changed. This would greatly speed up file crawling.
agarcian
Posts: 15
Joined: Wed Oct 20, 2004 2:07 pm

fine tuning performance.

Post by agarcian »

Yes, I am using file: protocol

I think the If-Modified-Since would be an incredible addition to the tool. That is exactly what I am looking for.

I thought about searching across profiles because I think it would be beneficial to split the documents into several profiles because of the performance of the crawl not the performance on the search.

I assume that there is no such cross-profile search allowed, right?
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

fine tuning performance.

Post by mark »

An update scheduled for release sometime next week has meta search for searching across profiles and/or across multiple appliances.
Post Reply