Open shared mem failed, No space left on device

aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

Open shared mem failed, No space left on device

Post by aitchon »

Would running rmlocks periodically to test if the shared mem error is happening be ok? I'm seeing these errors come up more often since I'm creating/deleting more temp db's. I'm thinking about scheduling the following script every hour:

if rmlocks -f 2>&1| grep -q "Open shared mem failed"
then
echo "failed"
ipcs -m | awk '/^0x/ && $6 == 0 && $1 != "0xdbaccee5" {print $2}' | while read id; do ipcrm -m $id; done
else
echo "pass"
fi
User avatar
Kai
Site Admin
Posts: 1271
Joined: Tue Apr 25, 2000 1:27 pm

Open shared mem failed, No space left on device

Post by Kai »

It would be better to rmlocks -f each temp db when you are done with it (i.e. just before deletion), to free its shared mem. That way no stray segments should be left around to cause the problem in the first place.

Wholesale automatic deletion of shared mem/lock structures may cause problems. Ideally just the oldest segments could be deleted, enough to keep the shared mem count under the max. You can get the timestamp of last access for a shmem segment with ipcs -m -t, but parsing that in a script (and getting the key) is non-trivial. In any event, rmlocks -f of each db as it is deleted should work better.
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

Open shared mem failed, No space left on device

Post by aitchon »

I'm still having issues with these temporary databases. I get this error when trying to create a new db:

-bash-4.1$ creatdb ~/tempdb
002 Could not obtain semaphore: No space left on device in the function initmutex
002 Texis Monitor message: Could not obtain semaphore: No space left on device in the function initmutex
000 Could not create locks for database /mnt/data/ftp/tempdb via monitor: Unsuccessful response code 500 (Failed) in the function TXcreateLocksViaMonitor
000 Could not open locking mechanism for database /mnt/data/ftp/tempdb in the function TXdblockOpenViaMethods

Is there a way to get a list of all active databases? I remember you told me ipcs -s | grep ^0x will give me the semid's for all db. I was thinking I could go though that list and do a rmlocks -f on each inactive db. Sort of the reverse of an ltest -d call that would return active databases.

Is it even possible to remove this semaphore error without rebooting? I've tried this but it doesn't seem to have affect:

ipcs -m | awk '/^0x/ && $6 == 0 && $1 != "0xdbaccee5" {print $2}' | while read id; do ipcrm -m $id; done
User avatar
Kai
Site Admin
Posts: 1271
Joined: Tue Apr 25, 2000 1:27 pm

Open shared mem failed, No space left on device

Post by Kai »

There's no current way to list active databases in Texis unfortunately. That ipcs command line will remove idle shared mem segments; in this case you're running out of semaphores, so while it probably freed up some shared mem, it didn't address the immediate issue (lack of semaphores).

Try this (as the Texis user):

ipcs -s | awk '/^0x/ {print $2}' | while read semId; do ipcrm --semaphore-id "$semId" done

Note that this could break locks on *live* databases as well, so stop any Texis activity (especially insert/delete/update/modify etc. activity) to avoid potential db corruption.
User avatar
mark
Site Admin
Posts: 5513
Joined: Tue Apr 25, 2000 6:56 pm

Open shared mem failed, No space left on device

Post by mark »

Semaphores are totally different things than shared mem.
"ipcs -s" to see semaphores, "ipcs -m" to see shared mems.
Other things on the system might be using either. You'll only want to be deleting those owned by your texis user.

You should do rmlocks -f on any database before deleting it to make sure it's resources are cleaned up.

No reboots required to delete shared mem or semaphores.
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

Open shared mem failed, No space left on device

Post by aitchon »

Kai, I tried the ipcs command and it looked like it worked, but I still get errors:

-bash-4.1$ creatdb ~/tempdb
100 Open shared mem failed: No space left on device; trying again in the function TXgetLockObject
100 Open shared mem failed: No space left on device; trying again in the function TXgetLockObject
100 Open shared mem failed: No space left on device; trying again in the function TXgetLockObject
000 Open shared mem failed: No space left on device in the function TXgetLockObject
002 Cannot open shared mem /mnt/data/ftp/tempdb: No space left on device in the function TXgetLockObject
005 Timeout reading data from 127.0.0.1:10005 in the function htbuf_readnblk
000 Could not create locks for database /mnt/data/ftp/tempdb via monitor: Connection timeout in the function TXcreateLocksViaMonitor
000 Could not open locking mechanism for database /mnt/data/ftp/tempdb in the function TXdblockOpenViaMethods
004 Cannot create directory /mnt/data/ftp/tempdb: Directory exists in the function TXcreateDb
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

Open shared mem failed, No space left on device

Post by aitchon »

I did modify your ipcs -m command by running:

ipcs -m | awk '/^0x/ {print $2}' | while read id; do ipcrm -m $id; done

That seemed to work. I was able to creatdb without error. The only activity on these databases are inserts/reads. Will this have any effect on active databases?
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

Open shared mem failed, No space left on device

Post by aitchon »

Actually, I tried on another server and got this error:

-bash-4.1$ creatdb ~/tempdb
000 Texis Monitor process PID 21015 failed (still running)
100 License Violation in the function TXcreateDb
User avatar
Kai
Site Admin
Posts: 1271
Joined: Tue Apr 25, 2000 1:27 pm

Open shared mem failed, No space left on device

Post by Kai »

Modifying the ipcs -m command by removing the 0xdbaccee5 check will flummox the Texis monitor. Try restarting the Texis monitor with:

monitor -k && /usr/local/morph3/bin/monitor

An insert is a db modification, so running either of the ipcs -s or ipcs -m commands -- which forcibly remove locks structures -- could cause db corruption if inserts/deletes/updates or any other mod is going on while they are run. Stop all db modifications before running those commands. And use rmlocks -f on known, idle databases to free up semaphores/shared mem before resorting to those ipcs commands.
aitchon
Posts: 118
Joined: Mon Jan 22, 2007 10:30 am

Open shared mem failed, No space left on device

Post by aitchon »

I tried this but the shared mems are still there:

ipcs -m | awk '/^0x/ && $1 != "0xdbaccee5" {print $2}' | while read id; do ipcrm -m $id; done
Post Reply