Thunderstone Support Forums

Hi,
We have a table which is rather large. We had put one of the fields that is huge (OCR like) into a blob type in this table. The current size of the table is,
2.3 GB for the .TBL file
41.9 GB for the .BLB file

When we tried to create the index
one virtual index (called idxmtbldiscovery) on all the fields in the table (including the blob) and

the index (called idxmtbldiscovery_DOCTEXT) on the blob field by itself

we got the following errors,

011 Out of memory in the function putkloc
000 Abandoning index /database03/c3blue/tbldiscovery/idxmtbldiscovery due to
errors
000 SQL failed
011 Can't alloc 311571255 bytes of memory: Resource temporarily unavailable in
the function fdbi_allocbuf
000 Abandoning index /database03/c3blue/tbldiscovery/idxmtbldiscovery_DOCTEXT
due to errors
000 SQL failed


any reason why? How to solve it?
We are running, Texis on UNIX/solaris 2.8. The machine has 2GB of RAM, 2 450 ultra processors, 2GB of swap space (I think).

Thanks!!

What version of texis is this (texis -version)? What are your current resource limits (ulimit -a)?

Hi,

Texis version
Texis Web Script (Vortex) Copyright (c) 1996-2001 Thunderstone - EPI, Inc.
Commercial Version 3.01.984591953 of Mar 14, 2001 (sparc-sun-solaris2.6)

resource limits, (ulimit -a)

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) unlimited
nofiles(descriptors) 256
memory(kbytes) unlimited

Thanks

Texis versions prior to March 2002 required more memory for write buffering, especially with large indexes. Also, with the Solaris 2.6 version memory access is limited to 32 bits, despite 64-bit file access, so in trying to use the default 40% of physical RAM for indexingTexis may be exceeding the OS address space overall.

You can reduce the amount of memory used for indexing to free up memory for the write buffer by setting indexmem to a lower value in SQL:

set indexmem='250MB'; create metamorph .....;

250MB is a starting point; less will free up more memory at the cost of extra file handles needed. (Overall index create speed should not be affected much.)

Thanks, I tried indexmem='250MB' with my index statements and I got the following errors,

002 Can't create exclusive KDBF file
/database03/c3blue/tbldiscovery/T28398lt.dat for writing: error 24: Too many
open files
002 Can't open /database03/c3blue/tbldiscovery/T28398lt.dat: Too many open
files in the function openwtix
000 Abandoning index /database03/c3blue/tbldiscovery/idxmtbldiscovery due to
errors
000 tsql ABEND: signal 11
002 Can't create exclusive KDBF file
/database03/c3blue/tbldiscovery/T28399lt.dat for writing: error 24: Too many
open files
002 Can't open /database03/c3blue/tbldiscovery/T28399lt.dat: Too many open
files in the function openwtix
000 Abandoning index /database03/c3blue/tbldiscovery/idxmtbldiscovery_DOCTEXT
due to errors
000 tsql ABEND: signal 10



thanks

Kai implied that that might happen and suggested varying the indexmem. Now try something between 250MB and 800MB to see where the balance point is for your system. Or increase the file handle ulimit.

Hi, I tried upto 750MB in increments of 50MB starting from 250MB. I got the errors too many file handles open immediately. I set the ulimit to 4096 for file handles. Still the same problem.
am I doing something wrong?
finally, I did not use indexmem at all, left it as the default, but set the file handles ulimit to 4096 and it looked like it ran for a long time but got the first error again,

Out of memory in the function putkloc
000 Abandoning index /database03/c3blue/tbldiscovery/idxmtbldiscovery due to
errors
000 SQL failed
011 Can't alloc 311571255 bytes of memory: Resource temporarily unavailable in
the function fdbi_allocbuf

any ideas?

My fault; the `MB' syntax wasn't support yet in your version. Try:

set indexmem=250000000; create metamorph .....

and if that fails, try 200000000, etc.