We're using the Linux version of Texis, release # 4.04.1067366033.
We have a 185 GB drive partition, on which a database sits. Right now the table, blob file and indexes take up 130 GB.
When we try to index the (by far) largest field in the table we eventually get an error that reads, "006 Can't write 0x20000 bytes at 0x2EC170E38 to KDBF file /database/558/mytable/mytable_DOCTEXT.dat: error 28: No space left on device in the function write_outbuf
".
We estimate that when the indexing is done this field's index will be ~15GB in size.
When the indexing died out it left behind four T*.dat files (and four associated .btrs). The .dats were all in the 900MB range, which from I've seen is the typical Texis pattern before starting the next "a.dat, b.dat, c.dat...." in the series.
So....we have 55 GB free at the moment on the volume, which I'd think is enough to get the job done? We looked at our tmp directory, but that's not filled up by any means.
Could you think of what's happening? If you think it's a disk space issue due to temp files we can expand the 185 GB to something larger.
It's a disk space issue, since the "No space left on device" message occurs when trying to write to a KDBF file.
After creation Metamorph inverted indexes typically take around 30% of indexed-field(s) data size, ie. 30% of table size in this case. However that can vary depending on the data. Also, up to another 30% is needed for temp files during the build. So free space should be at least 60% of table size. Ie. with a 100GB table, you should expect to need around 60GB of free space to build an index.
Note that you can spread the final Metamorph index, temp files, and the original table onto different partitions (if your filesystem is not dynamically re-partitionable) with the indexspace, addindextmp, and tablespace settings. For example, during the index build, you could use other filesystem(s) for temp index files:
set addindextmp='/usr5/tmp';
set addindextmp='/usr6/tmp';
set addindextmp='/usr7/tmp';
create metamorph inverted index ...
This would use /usr5/tmp, /usr6/tmp and /usr7/tmp for temp index files, instead of the index directory.
Could I ask one more question? The reindex takes most of the CPU's resources, not surprising given its size.
It would be ideal to throttle it down some, so that it maxed out at, say, 50% of processor utilization. I _believe_ there's a flag I can set to do that in my tsql command? Or am I thinking of something else?
You can't directly throttle the CPU usage. If nothing else is using that database you could use the OS "nice" command so the index build will be lower priority, and only use the CPU when other processes aren't. Do not use that if you are likely to have other processes locking the tables, as the higher priority tasks may deadlock out the lower priority trying to get a lock.
Hmm....I ran this command as a test, against a big table of ours.
nice -19 tsql "select count(ID) from mytable where AUTHOR like 'james'"
I didn't see any noticeable improvement in the CPU utilization. It still used 50% of the CPU resource, just as before.
From what I've read the 19 should kick this to the lowest possible priority. Maybe I've typed it wrong or misunderstood the parameters to pass in? Wouldn't be the first time.
By specifying the nice it lowers the priority, but if no other process wants to use the CPU it can still use the entire CPU. I don't think there is much reason to not use the entire CPU if no other process wants it.
You're right. The same thought crossed my mind about five seconds after I posted. No point in a process not taking all the CPU if no one else is in line.
