Locking showing up on 2 databases

MiniMe · Post by **MiniMe** » Wed Jul 16, 2008 10:38 am

I have Texis running on a Linux server. There are two databases. One is for maint and the other is production. Recently a weird load issue has shown up. After some investigation I discovered that if you run ltest on /db/tables/maint and let it sit for a few moments it suddenly changes to /db/tables/prod and shows the same locking that was going on in maint. If I go to /db/tables/prod I can see the locking from the maintenance routine since the PIDS match the script that is running. Also the monitor is shown with the SAME PID on both databases. What gives??

Post by **mark** » Wed Jul 16, 2008 10:50 am

Are your SURE they are different? cd to each one in turn and run "/bin/pwd" to see what you get.

cd /db/tables/maint
/bin/pwd
cd /db/tables/prod
/bin/pwd

MiniMe · Post by **MiniMe** » Wed Jul 16, 2008 10:52 am

Of course.. ;>)

[root@jetsam prod]# cd /db/tables/maint/
[root@jetsam maint]# pwd
/db/tables/maint
[root@jetsam maint]# cd /db/tables/prod
[root@jetsam prod]# pwd
/db/tables/prod
[root@jetsam prod]#

Post by **John** » Wed Jul 16, 2008 11:16 am

Do you have something that switches the databases by renaming the directories? That might cause the change in database that ltest shows, as it will stay with the database as it is renamed, and any running processes, such as the maintenance process, would stay on the same database as it is renamed.

MiniMe · Post by **MiniMe** » Wed Jul 16, 2008 11:24 am

We don't rename the directories. We do a file move of all of the non-system tables and indexes in the middle of the night. We have used the same methodology for years to migrate from maint to prod so I am not sure how this would suddenly start impacting the database.

I tried to start monitor on /db/tables/maint just now but it says its already running but a ps -ef says no.

Post by **mark** » Wed Jul 16, 2008 11:39 am

Try "/bin/pwd" as "pwd" just prints what you cd'd to not where you really are if it's different.

Also do ls -l in both directories and make sure nothing's linked (hard or soft) between the 2.

ls -lid /db/tables/maint /db/tables/prod
ls -li /db/tables/maint /db/tables/prod

MiniMe · Post by **MiniMe** » Wed Jul 16, 2008 11:46 am

Everything looks good there. I even dismounted the slices to see if there was a connection and there was not. No symlinked files, no symlinked directories.

I also tried the tip on your site about deleting shared memory segments. No go there either. This is really strange. Does this antique version of texis not like the kernel or something?

Post by **John** » Wed Jul 16, 2008 12:27 pm

When you did the

ls -lid /db/tables/maint /db/tables/prod

where the inodes different for each one, and for the SYSTABLES in each database? If the device and inode number look the same then that could result in confusion.

Which version of Texis and which Linux kernel are you using, as if the stat() call is truncating data that could also be an issue.

MiniMe · Post by **MiniMe** » Wed Jul 16, 2008 12:31 pm

Here is the output of the SYS files..
[root@jetsam /]# ls -lid /db/tables/maint/SYS* /db/tables/prod/SYS* |sort -n
12 -rw-r--r-- 1 database root 9902 Jul 16 08:06 /db/tables/maint/SYSTABLES.tbl
13 -rw-r--r-- 1 database root 67876 Jul 16 08:06 /db/tables/maint/SYSCOLUMNS.tbl
14 -rw-r--r-- 1 database root 9598 Jul 16 08:06 /db/tables/maint/SYSINDEX.tbl
15 -rw-r--r-- 1 database root 3446 Dec 6 2006 /db/tables/maint/SYSUSERS.tbl
16 -rw-r--r-- 1 database root 5484 Jul 16 08:06 /db/tables/maint/SYSPERMS.tbl
17 -rw-rw-rw- 1 database root 8 Jul 16 08:44 /db/tables/maint/SYSLOCKS.SEQ
18 -rw-r--r-- 1 database root 3322 Dec 6 2006 /db/tables/maint/SYSTRIG.tbl
19 -rw-r--r-- 1 database root 3322 Dec 6 2006 /db/tables/maint/SYSMETAINDEX.tbl
19 -rw-rw-rw- 1 database root 8 Jul 16 09:29 /db/tables/prod/SYSLOCKS.SEQ
20 -rw-r--r-- 1 database root 9620 Jul 16 07:22 /db/tables/maint/SYSSTATS.tbl
24 -rw-r--r-- 1 database root 131214 Jul 2 06:14 /db/tables/prod/SYSCOLUMNS.tbl
25 -rw-r--r-- 1 database root 24266 Jul 16 02:22 /db/tables/prod/SYSINDEX.tbl
27 -rw-r--r-- 1 database root 3322 Dec 1 2006 /db/tables/prod/SYSMETAINDEX.tbl
28 -rw-r--r-- 1 database root 6044 Jul 2 06:14 /db/tables/prod/SYSPERMS.tbl
29 -rw-r--r-- 1 database root 14446 Jul 16 09:27 /db/tables/prod/SYSSTATS.tbl
30 -rw-r--r-- 1 database root 13310 Jul 2 06:14 /db/tables/prod/SYSTABLES.tbl
31 -rw-r--r-- 1 database root 3322 Dec 1 2006 /db/tables/prod/SYSTRIG.tbl
32 -rw-r--r-- 1 database root 3446 Dec 1 2006 /db/tables/prod/SYSUSERS.tbl

It is texis version 4.02.1033632267 with the linux kernel version 2.6.9-42.0.8.ELsmp

Post by **mark** » Wed Jul 16, 2008 1:16 pm

Looks like prod and maint are the mount points of different filesystems and therefore have the same inode number. Confirm with
ls -lid /db/tables/maint /db/tables/prod
If they are the same that would be causing collision of the shared mem segment key (on linux only). If that's the case try creating a new directory under the mount point for one of them so the inode of the db dir is different.