can't find required libraries libm, libcrypt, libc

[bblodgett]

Our webinator search engine suddenly quite working after some cgi content was downloaded an pushed back up to our web server. I've went through and verified permissions and ensured that the texis binary (which seems to be the crux of the problem is owned by the web server user and is also suid for this user) but we still get the following error when the texis binary attempts to run:

/lib/libcrypt.so.1: version `GLIBC_2.0' not found (required by texis)
/lib/libm.so.6: version `GLIBC_2.1' not found (required by texis)
/lib/libm.so.6: version `GLIBC_2.0' not found (required by texis)
/lib/libc.so.6: version `GLIBC_2.1' not found (required by texis)
/lib/libc.so.6: version `GLIBC_2.0' not found (required by texis)

Those libraries do indeed exist in /lib (as symlinks to the true library versions) and ldconfig has been run several times to no avail. I understand this is probably related to some sort of permissions issue but I've tried changing the binary user and group ownerships to everthing under the sun to no avail and the error messages continue to persist.

[mark]

Wow, you really did a number to your system. Are you sure the binaries didn't get corrupted in either download or upload? Permissions don't matter much until the program and the core libraries are loaded. Just that the binary and libs must be accessable by the user attempting to execute the binary. Are the libs accessable by the web server user?

Are you perhaps running the webserver or CGIs in a chrooted environment where the libs may be different than otherwise?

Does "texis" work from the shell prompt (try texis -version)?

[bblodgett]

The odd thing here is that the only thing that was ftped back up that was related to webinator was the texis binary, and the fact that it actually gave me the can't find libraries error rather than just segfaulting (which is usually what happens if you try running a binary you have pushed as ascii vi ftp) puzzles me.

There is no chroot environment for the web server root or cgi root on this web server.

Trying to run texis from the command line returns the same unable to find libraries errors.

[mark]

Are you sure those library soft links point to actual glibc 2 libraries? Sounds like maybe they point to something else that doesn't have a proper GLIBC_2 tag.

BTW, why did you download and reupload? System maintenance of some kind? Or system transfer? I'm really suspecting a system level change of some kind at this point.

[bblodgett]

Here is the library specific info:

/lib/libm.so.6 -> libm-2.1.2.so
/lib/libm-2.1.2.so: ELF 32-bit LSB shared object, Intel 80386, version 1, not stripped

/lib/libcrypt.so.1 -> libcrypt-2.1.2.so
/lib/libcrypt-2.1.2.so: ELF 32-bit LSB shared object, Intel 80386, version 1, not stripped

/lib/libc.so.6 -> libc-2.1.2.so
/lib/libc-2.1.2.so: ELF 32-bit LSB shared object, Intel 80386, version 1, not stripped

The web content guy was changing fields in a large number of cgi scripts and opted to download the entire cgi tree and push it back up, this was the only thing that changed since webinator quit working (i.e. everything else at the system level is exactly the same, to include all the webinator pieces except for texis which resided in the cgi directory).

[mark]

Try these commands to see if they give any more info about the error:
ldd texis
strace texis

[bblodgett]

On a side note I'll mention that I copied the texis binary to another machine with an identical Redhat install/version just to see if I could get it to run and ended up getting the same error that is indicated by the ldd output.

Here is the output of those two commands (actually I'd been running ldd to get the library deps initially):

# ldd texis
./texis: /lib/libcrypt.so.1: version `GLIBC_2.0' not found (required by ./texis)
./texis: /lib/libm.so.6: version `GLIBC_2.1' not found (required by ./texis)
./texis: /lib/libm.so.6: version `GLIBC_2.0' not found (required by ./texis)
./texis: /lib/libc.so.6: version `GLIBC_2.1' not found (required by ./texis)
./texis: /lib/libc.so.6: version `GLIBC_2.0' not found (required by ./texis)
libm.so.6 => /lib/libm.so.6 (0x40017000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x40034000)
libc.so.6 => /lib/libc.so.6 (0x40061000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)


# strace /usr/local/apache/docroot2/cgi-bin/texis
execve("/usr/local/apache/docroot2/cgi-bin/texis", ["/usr/local/apache/docroot2/cgi-bin/texis"], [/* 19 vars */]) = 0
brk(0) = 0x84ce85c
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=14394, ...}) = 0
mmap(0, 14394, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40013000
close(3) = 0
open("/lib/libm.so.6", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=540120, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320=\0"..., 4096) = 4096
mmap(0, 114648, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40017000
mprotect(0x40032000, 4056, PROT_NONE) = 0
mmap(0x40032000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1a000) = 0x40032000
close(3) = 0
open("/lib/libcrypt.so.1", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=64595, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\f\0"..., 4096) = 4096
mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40033000
mmap(0, 183036, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40034000
[root@gorgo /root]# strace /usr/local/apache/docroot2/cgi-bin/texis
execve("/usr/local/apache/docroot2/cgi-bin/texis", ["/usr/local/apache/docroot2/cgi-bin/texis"], [/* 19 vars */]) = 0
brk(0) = 0x84ce85c
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=14394, ...}) = 0
mmap(0, 14394, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40013000
close(3) = 0
open("/lib/libm.so.6", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=540120, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320=\0"..., 4096) = 4096
mmap(0, 114648, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40017000
mprotect(0x40032000, 4056, PROT_NONE) = 0
mmap(0x40032000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1a000) = 0x40032000
close(3) = 0
open("/lib/libcrypt.so.1", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=64595, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\f\0"..., 4096) = 4096
mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40033000
mmap(0, 183036, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40034000
mprotect(0x40039000, 162556, PROT_NONE) = 0
mmap(0x40039000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x40039000
mmap(0x4003a000, 158460, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4003a000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=4118299, ...}) = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\250\202"..., 4096) = 4096
mmap(0, 993500, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40061000
mprotect(0x4014c000, 30940, PROT_NONE) = 0
mmap(0x4014c000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xea000) = 0x4014c000
mmap(0x40150000, 14556, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40150000
close(3) = 0
write(2, "/usr/local/apache/docroot2/cgi-b"..., 40/usr/local/apache/docroot2/cgi-bin/texis) = 40
write(2, ": ", 2: ) = 2
write(2, "/lib/libcrypt.so.1", 18/lib/libcrypt.so.1) = 18
write(2, ": ", 2: ) = 2
write(2, "version `GLIBC_2.0\' not found (r"..., 84version `GLIBC_2.0' not found (required by /usr/local/apache/docroot2/cgi-bin/texis)) = 84
write(2, "\n", 1
) = 1
write(2, "/usr/local/apache/docroot2/cgi-b"..., 40/usr/local/apache/docroot2/cgi-bin/texis) = 40
write(2, ": ", 2: ) = 2
write(2, "/lib/libm.so.6", 14/lib/libm.so.6) = 14
write(2, ": ", 2: ) = 2
write(2, "version `GLIBC_2.1\' not found (r"..., 84version `GLIBC_2.1' not found (required by /usr/local/apache/docroot2/cgi-bin/texis)) = 84
write(2, "\n", 1
) = 1
write(2, "/usr/local/apache/docroot2/cgi-b"..., 40/usr/local/apache/docroot2/cgi-bin/texis) = 40
write(2, ": ", 2: ) = 2
write(2, "/lib/libm.so.6", 14/lib/libm.so.6) = 14
write(2, ": ", 2: ) = 2
write(2, "version `GLIBC_2.0\' not found (r"..., 84version `GLIBC_2.0' not found (required by /usr/local/apache/docroot2/cgi-bin/texis)) = 84
write(2, "\n", 1
) = 1
write(2, "/usr/local/apache/docroot2/cgi-b"..., 40/usr/local/apache/docroot2/cgi-bin/texis) = 40
write(2, ": ", 2: ) = 2
write(2, "/lib/libc.so.6", 14/lib/libc.so.6) = 14
write(2, ": ", 2: ) = 2
write(2, "version `GLIBC_2.1\' not found (r"..., 84version `GLIBC_2.1' not found (required by /usr/local/apache/docroot2/cgi-bin/texis)) = 84
write(2, "\n", 1
) = 1
write(2, "/usr/local/apache/docroot2/cgi-b"..., 40/usr/local/apache/docroot2/cgi-bin/texis) = 40
write(2, ": ", 2: ) = 2
write(2, "/lib/libc.so.6", 14/lib/libc.so.6) = 14
write(2, ": ", 2: ) = 2
write(2, "version `GLIBC_2.0\' not found (r"..., 84version `GLIBC_2.0' not found (required by /usr/local/apache/docroot2/cgi-bin/texis)) = 84
write(2, "\n", 1
) = 1
_exit(1) = ?

[bblodgett]

Mark,

Hmmm, this one ended up being pretty simple, it looks like the texis binary got mangled in some manner that made it appear to be intact (i.e. I was still able to get information out of it via ldd, strace, etc) but it didn't want to run.

I extracted the texis binary from the webinator tarball I downloaded from your site a few minutes ago and dropped it into place on the web server and the search engine appears to be working properly now.

[mark]

All I can find is that copying a newer glib over without also ld-linux can cause this kind of thing (but that might mess up the whole system).
What's the output of:
nm /lib/ld-linux.so.2 | grep GLIBC
And the "Version definitions" section of:
objdump -p /lib/libc.so.6

[mark]

I guess it took me too long to compose that last msg. Didn't see yours til i submitted it. Very strange indeed.