Wrong server id

Post by **mark** » Thu Feb 21, 2002 12:03 pm

The cookies thing is from overagressive cache closes. In the script right AFTER "create ram table cookies" (line 535) delete the line <sqlcp cache close>.

You may be hitting a memory limit or a ulimit of some kind during index creation. In the "createindex" function (around line 850 of dowalk) add the following line AFTER the <sqlcp cache close> line:
<sql "set indexmem=25"></sql>

BTW, how much total ram does the system have? If Texis has a problem determining the amount that could cause a problem. In that case you can tell it how much to use with something like:
<sql "set indexmem=50000000"></sql>

Post by **mark** » Thu Feb 21, 2002 12:04 pm

It might expedite matters if we could access the machine. If you would like us to do so, please open a tech support ticket at http://www.thunderstone.com/texis/site/ ... pport.html

Post by **mark** » Thu Feb 21, 2002 12:16 pm

Also check your ulimits (ulimit -a). they may be too small therefore artificially restricting the system mem available for Texis. The default appears to be 64 meg which may be small for large index creation.

You can set indexmem low, like 30 meg (30000000) to work within that space if needed.

jai.thomas · Post by **jai.thomas** » Thu Feb 21, 2002 12:48 pm

I have manually did a create index using tsql and it worked with indexmem=50000000. Now, will the memory requirement for indexing go up wehen there are more pages. We are yet to walk all sites in our list.

SQL 1>select count(*) from html;
count(*)
------------+
18427
SQL 1>drop index xhtmlbodv;
SQL 1>set indexmem=25;
SQL 1>create metamorph inverted index xhtmlbodv on html(Title\Description\Keywor
ds\Meta\Body,Visited);
011 Can't alloc 1966088 bytes of memory: Not enough space in the function newchu
nk
000 Abandoning index /appsvr/texsvr/usr/local/morph3/texis/agsites/db1/xhtmlbodv
due to errors
000 SQL failed
SQL 1>set indexmem=50000000;
SQL 1>create metamorph inverted index xhtmlbodv on html(Title\Description\Keywor
ds\Meta\Body,Visited);

I am not sure of providing access to the machine. It is inside firewall and I need to check with admin folks.

Now, All these are done on our test server. We configured Texis on production server exactly as on test and installed latest files. But I am seeing something weird there. When I create the profile, it just waits and do not come back. But profile directories are creeated and when I refresh screen profile name is listed as normal. But, when I start a walk, it is not initiating a walk, Status doesn't say much and just displays the following:

Walk Status
Current Profile: AGSITES Webinator 4.0.2
Site not submitted for walking yet.
Go to Walk Settings to configure and/or start a walk.

ps -elf list the walk process in sleep state like this
1 S www 4972 1 55 168 24 48841100 829 400003ffffff0000 10:45:12 ?
0:01 ./texis profile=AGSITES /appsvr/inet/lyport/web/webinator/do

Any idea what is wrong here? Production server has similar configurationas test, just that it is a bigger box.

Thanks
Jai

jai.thomas · Post by **jai.thomas** » Thu Feb 21, 2002 1:03 pm

Looks like -a is not a good ulimit argument.
$ ulimit -a
ksh: ulimit: bad option(s)
$ ulimit
4194303

Both of our servers have 2GB memory and more than 25GB free disk space.

Thans
Jai

Post by **mark** » Thu Feb 21, 2002 1:09 pm

So you are hitting the ulimit. Put the working indexmem adjustment in the script or increase your system ulimit. You should also put the same thing right after the <sqlcp cache close> in the updatemmindex function.

If you click "index status" again does it then show status normally? If so, increasing the value of startsleep in osinfo from 2 to 4 or 5 might make that work more smoothly.

Post by **mark** » Thu Feb 21, 2002 1:12 pm

Maybe you're using a nonstandard shell. The standard /bin/sh on our system respects ulimit -a . In any event you're probably hitting the system default limit of 64meg per process (based on your indexmem test results).

jai.thomas · Post by **jai.thomas** » Thu Feb 21, 2002 1:24 pm

You are right about sh.
Here we go
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 65536
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 4096

What is the recommended ulimit?

In 26, I assume you are refering to 'Walk Status'.
Clicking it again doesn't show anything different.
In fact walk settings and status says 'Site not submitted for walking yet.'

This behavior has started only after installing new script files. Before it used to attempt walk right away.
How do I see startsleep value?

Thanks
Jai

Post by **mark** » Thu Feb 21, 2002 2:31 pm

A couple hundred megs for data would be better, but less is workable. The problem is that the index tries to use a percentage of PHYSICAL ram by default or if you give a number less than 100. That doesn't work well when the ulimit is less than total ram size. Any time the ulimit is less than phyisical ram you'll need to set indexmem to some number under the ulimit.

startsleep is hardcoded into the dowalk script. Look for startsleep= (there're 2, one for NT and one for unix). It doesn't appear to be the problem though if clicking walk status doesn't help.

Did you use the same scripts from your test server?
Did you make sure all shared mems and such from previous failures were cleaned up first?
Check /usr/local/morph3/texis/monitor.log and vortex.log to see if there's anything interesting there.

jai.thomas · Post by **jai.thomas** » Thu Feb 21, 2002 2:46 pm

script and configuration is same.
shared mems were cleaned up prior to installing the new scripts but not after that. Should I do it once again?
Monitor and vortex logs do not report any errors.
Thanks
Jai