Large indexes and live search

scott.shaver · Post by **scott.shaver** » Fri Mar 17, 2006 2:32 pm

Thanks very much for your help guys.

scott.shaver · Post by **scott.shaver** » Mon Mar 20, 2006 11:03 am

Okay so this thing is still running after 4 or 5 days now. Very few new "pages" have been added to the count since Friday. I can see it moving through the directories and files. Why is it taking so long, and why is the page count not going up?

and the CPU usage is only like 4%

Post by **John** » Mon Mar 20, 2006 11:50 am

Are there actually more files it should be indexing? It almost sounds as if it might have found all the documents, and is now in refresh mode.

Post by **mark** » Mon Mar 20, 2006 11:55 am

"few" meaning roughly how many?
In what way do you see it "moving through"?
You don't have a crawl delay of anything other than 0 do you?
Is the connection between the appliance and fileserver a fast one?

scott.shaver · Post by **scott.shaver** » Mon Mar 20, 2006 12:06 pm

"Are there actually more files it should be indexing?"

Well I can't tell, it seems to be going through the directories in alphabetical order. If it really is then there are a lot of files to go.

"In what way do you see it "moving through"?"

by whatching the file url list in the walk status page, it changes every now and again.

""few" meaning roughly how many?"

about 300

"You don't have a crawl delay of anything other than 0 do you?"

nope it is zero

"Is the connection between the appliance and fileserver a fast one?"

It isn't an appliance it's a texis installation on a very nice server. The network is quite fast. When the index first started it was flying throught the files very fast.

Post by **mark** » Mon Mar 20, 2006 12:18 pm

How large are the texis processes?
Does it speedup if you "pause and live" then "go" (in mode refresh) again?

scott.shaver · Post by **scott.shaver** » Mon Mar 20, 2006 12:27 pm

I have one texis process that is current 90 megs and 2 monitor processes that are about 4.3 meg each.

I'll try pausing the walk and see what happens.

scott.shaver · Post by **scott.shaver** » Mon Mar 20, 2006 1:16 pm

How long does it normally take to pause a walk? I know I'm not being patient but it would sure be nice to know the thing isn't hung

Walk started at 2006-03-17 12:05:53 (by resume)
JavaScript walking enabled
HTTPS walking disabled
Start fetching at file://evergreen/corp/
Ignore urls containing any of the following:
/cgi-bin/
~
?
/private
started 1 (3160) Resume 44187392f2
Walker holding by request. (file://evergreen/corp/)
8058 pages fetched (-1,511,690,516 bytes) from file://evergreen/corp/
started 1 (3192) Resume 44187392f2
Walker holding by request. (file://evergreen/corp/)
3 pages fetched (5,292,935 bytes) from file://evergreen/corp/
started 1 (3032) Resume 44187392f2
Show Errors

Post by **mark** » Mon Mar 20, 2006 2:07 pm

The walk status should say "stopping by request" which is usually quick (well under a minute) unless there's a zillion urls in memory to write out to disk for resumption later. Then it'll go into a "creating search index" phase which could take a fair number of minutes for a large dataset.

It looks like your walk started up again after being paused. You don't have it on a rapid schedule do you? If it doesn't go into the indexing phase within 30 seconds or so click the pause button again.

If you have remove common on that will happen before the indexing and could take a while on a large dataset.

scott.shaver · Post by **scott.shaver** » Mon Mar 20, 2006 2:20 pm

That restart was from friday. I don't have remove common on. The behavior you are describing is not what I'm seeing.

...

gak it just fried on me again

-----------------------------------------

Texis ISAPI
Texis ISAPI has been installed. However, it has encountered an error and cannot continue. Please check the event log for details on the problem.
-----------------------------------------

Texis ISAPI encountered a socket error:

Socket connection to remote Texis failed!

Please ensure that the host ((null)) and port (10700) are configured properly, and check "Texis/monitor.log" to see if the Monitor Web Server is running.

WSAGetLastError: 10061

-----------------------------------------
had to restart the monitor service.

The walk is dead now:

-----------------------------------------
Walk started at 2006-03-17 12:05:53 (by resume)
JavaScript walking enabled
HTTPS walking disabled
Start fetching at file://evergreen/corp/
Ignore urls containing any of the following:
/cgi-bin/
~
?
/private
started 1 (3160) Resume 44187392f2
Walker holding by request. (file://evergreen/corp/)
8058 pages fetched (-1,511,690,516 bytes) from file://evergreen/corp/
started 1 (3192) Resume 44187392f2
Walker holding by request. (file://evergreen/corp/)
3 pages fetched (5,292,935 bytes) from file://evergreen/corp/
started 1 (3032) Resume 44187392f2

-----------------------------------------

I can't tell if it finished or not. I suspect it didn't.