NOT searches - OUT of memory

skalyanaraman · Post by **skalyanaraman** » Thu Oct 02, 2003 5:16 pm

Hi,
We are running searches of the form,

tsql "select count(DOCID) from tbldoc where DOCTEXT NOT like 'orbiter'"

This runs for a long time and we get a "OUT of memory error" and the machine goes off-line (this is on linux).

The machine is a linux machine with 1/2Gig of memory and this tsql is the only process running.
The version of texis is 4.03.1060800870 of Aug 13, 2003.

The same thing looks like is happening on solaris from the Apr 15, 2003 version. We started this query and it took a long time. But we killed it.

Any help will be greatly appreciated.

Thanks!!

Post by **John** » Thu Oct 02, 2003 9:06 pm

We'll look into it. It is currently more efficient in Texis to do a count(DOCID) and a count(DOCID) where DOCTEXT like 'orbiter' and subtract them (assuming indexes exist).

skalyanaraman · Post by **skalyanaraman** » Wed Oct 08, 2003 3:17 pm

Thanks!!
The problem is we just don't do the count, we also write the DOCID's to a file. And this is a generic functionality of our website. So, we have to change our code if we want to treat NOT searches as a special case.
We also found out that the same problem exists in our Apr 15, 2003 version on Solaris.

Any more info on this issue?

Do you think it will be a separate fix in a build for us?

Thanks for all the help in advance.

Post by **John** » Wed Oct 08, 2003 5:02 pm

We have tried to replicate it here without success. Is there anything else you can tell us about the table or query that might help? On about 5 million rows and about 2GB of data the process never grows above 5MB of RAM.

skalyanaraman · Post by **skalyanaraman** » Wed Oct 08, 2003 5:11 pm

The table has a BLOB field names DOCTEXT.
So, these are the sizes.

tbldoc.tbl - 430MB
tbldoc.blb - 4.4GB

So, the table is around 4.5GB.
Total number of rows - 517600

We are searching the DOCTEXT field. This field has been fully indexed.

Number of rows that is "like 'orbiter'" - 309

So, everything else is "not like 'orbiter'" - 517291 rows

This last search is the one that gives out of memory.

Hope this helps!!!

skalyanaraman · Post by **skalyanaraman** » Mon Oct 13, 2003 5:15 pm

Hi,
I was just wondering whether you were able to re-create the problem there with the information provided in the last post.

We have tested it on our side on different data and could replicate it. Do you think it is the structure of the data or any other parameters specific to us?

Thanks!!

Post by **John** » Mon Oct 13, 2003 6:31 pm

Yes, it does appear to be related to blobs, and memory fragmentation. We are trying to create a fix for that.

skalyanaraman · Post by **skalyanaraman** » Tue Oct 28, 2003 7:35 pm

Hi,
Just wondering what the status of the NOT search and memory fragmentation fix is.
Please let us know, if you have any info.

Thanks in advance for all the help!!

Post by **John** » Tue Oct 28, 2003 9:44 pm

The fix is live, and in all current distributions.

skalyanaraman · Post by **skalyanaraman** » Wed Oct 29, 2003 3:42 pm

Thanks.
Can we get a ftp site where we can download the Linux and Solaris versions of this fix?

Also, can we get electronic copied in CDs for linux and Solaris mailed to us? Who do I have to contact to request this mailing of the media?

Thanks!!