Hi when a walk is in progress, search perfmormance drops dramatically ...
- No walk in progress : Search time = immediate
- 1 walk in progress : Search time > 10 sec.
- 3 walks in progress : Search time = time out
Enabling fast result counts does not help a lot.
Process size is set to medium
How many servers are these walks indexing?
Is the profile being queried the same one being walked?
Is it refresh or new walk?
What if you set crawl delay to 1?
>> How many servers are these walks indexing?
The big one was indexing 6 servers the other 2 one each. Each one has 5 threads.
I decreased the big one to 2 servers and i still get time-outs (although less frequently) as soon as a second walk is running simultaneously.
>> What if you set crawl delay to 1?
Changing crawl delay did not change anything.
>> Is it refresh or new walk?
All the walks were 'refresh'.
>> Is the profile being queried the same one being walked?
Yes and no. The big one was most affected (having time-outs. The other have their response time going up from 1-2 secs to 10-20.
Also reduce threads to 2 or 1. Try a process size of small to see if it helps without causing too many walk restarts.
Does the big one take a long time to do the whole job? And are a lot of pages changed on each cycle? If so it may be beneficial to pause-and-live it in the middle then resume it.
The only thing that helped me significantly reduce the timeout was to reduce the "Maximum Load Average". However, this does not completely eradicate the problem...
I set thrads, servers down to one and process size to small without much result.
Here are the numbers from th walk status page :
9,020 pages in todo
240,652 pages scheduled to be refreshed
85 pages visited in the last hour (42 success/43 failed)
249,999 pages in index
The refresh walk now lasts about 2-5 minutes because of the license limit.
NB: I didn't change the 'load average' setting, because the help does not specify it's range.
To do some further testing i did the following setup
Profile A 250 000 pages, having the performance problem
Thread : 1 Server : 1 delay : 1, size : small
Profile B 120 000 pages, not having the problem
Thread : 5 Server : 1 delay : 0, size : medium
1 : Do a search returning several 1000 documents on both profiles (repeat with 3 different searches)
A result takes 5-10 seconds
B result is instantaneous
2 : Launch a refresh walk for both profiles
3 : Do the same searches again
A result takes 15-20 seconds
B result is still instantaneous
That's new. I thought the slowness was across the board when any crawl was running. It appears the problem is just the one profile whether it's crawling or not, so most of my suggestions are moot.
What non-default search settings are you using for profile A?
Mark :
Search setting differences between A and B an DEFAULT :
A B DEFAULT
Query Logging Yes Yes no
Search Timeout 60 NULL NULL
Abstract Length 180 180 NULL
Results per Page 25 25 10
Display no no null
Thunderstone logo
Show yes yes null
Advanced Search
Top HTML Custom Custom Default
Bottom HTML Custom Custom Default
Synonyms Phrase& Phrase& Phrase
allow synonym allow synonym recognition
only
Fast Yes Yes No
Result Counts
Word Forms Plural& Plural& Exact
possesifs possesifs match
John :
more or less ... i have
Updating search index ...WARNING: You are within 10% of your page limit, 249999.
Creating spell-checker dictionaries...Done.
Verifying usability of new walk.
Could it be that the index is not updated naymore due to licence limits?
