Wrapping command-line programs, part IV
The first essay of this series showed how to use Python's subprocess module to do a one-shot call of an external program. The second turned that interface into a coprocess for better performance, at the cost of greater complexity. To make my code work I needed to recompile the code and flush the output after every name. That option isn't always available so the third essay showed how to use pseudo-ttys to force the coprocess into line buffer mode.
I have another trick up my sleeve. Nearly all programs, including the OpenEye ones, use shared libraries. When a program starts the run-time loader looks at which libraries is needs and loads them as well. Here's a list of the ones used by mol2nam. (I've switched to a Linux machine for this essay which is why the prompts are different.)
[~/src]$ ldd $OE_DIR/bin/mol2nam
linux-gate.so.1 => (0xffffe000)
libz.so.1 => /lib/libz.so.1 (0x4002c000)
libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x4003d000)
libm.so.6 => /lib/tls/libm.so.6 (0x400fe000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40121000)
libc.so.6 => /lib/tls/libc.so.6 (0x4012a000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
[~/src]$
The ones I recognize are the compression library ("libz"), the
standard C++ library ("libstdc++"), the math library ("libm"), the gcc
library ("libgcc_s") and the standard C library ("libc").
Here's a list of some of the external functions mol2nam needs in order to run:
[~/src]$ objdump -T $OE_DIR/bin/mol2nam /usr/local/openeye/bin/mol2nam: file format elf32-i386 DYNAMIC SYMBOL TABLE: 00000000 DF *UND* 00000065 GLIBCPP_3.2 _ZNKSs17find_first_not_ofEPKcjj 00000000 DF *UND* 00000023 GLIBC_2.0 __umoddi3 00000000 DF *UND* 000000dc GLIBC_2.0 __divdi3 00000000 DF *UND* 000002fa deflate 00000000 DF *UND* 00002c59 GLIBC_2.0 __strtod_internal 00000000 DF *UND* 0000005b GLIBCPP_3.2 _ZNKSs7compareERKSs 00000000 DF *UND* 000000c6 GLIBC_2.0 vsprintf 00000000 w D *UND* 00000000 pthread_create 00000000 DF *UND* 000000a8 GLIBCPP_3.2 _ZNSs7replaceEN9__gnu_cxx17__norma l_iteratorIPcSsEES2_jc 00000000 DF *UND* 0000006d GLIBC_2.0 feof 082334c0 w DO .bss 00000014 GLIBCPP_3.2 _ZTVSt9bad_alloc 082334d8 w DO .bss 00000010 GLIBCPP_3.2 _ZTIPb 00000000 DF *UND* 000000f1 GLIBC_2.0 ungetc ...The complicated names are for the C++ function name mangling needed to work with a library format designed for C-style names.
With care and with some restrictions it's possible to tell the run-time loader to use functions from another library. The mechanism is different on different operating systems but the general idea is to create what's often call a shim. The shim implements one or more of the functions needed by the program so the loader imports those definitions. When the program calls that function the shim is then free to do whatever it wants. In some cases it may define completely new functionality, like a new memory allocator or random number generator. In others it may extend existing functionality like zlibc which transparently opens compressed files as if they are uncompressed. If the file is not compressed it forwards the call to the original function.
In this case I want to set stdout to be in line-buffered mode. I need to hook into a function mol2nam needs. I tried fdopen() and printf() but those didn't seem to work and I don't know why. Instead I remembered that the OpenEye tools use getenv() to get the value of the OE_DIR environment variable. That's an easy hook so I wrote the following as linebuffered_stdout.c:
/* LD_PRELOAD shim to set stdout to line buffered mode when
getenv is called */
/* "The symbols RTLD_DEFAULT and RTLD_NEXT are defined by <dlfcn.h>
only when _GNU_SOURCE was defined before including it. " */
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>
char *getenv(const char *s) {
char *(*real_getenv)(const char *s);
setlinebuf(stdout);
real_getenv = dlsym(RTLD_NEXT, "getenv");
return real_getenv(s);
}
The function has the same call signature as the real
getenv(). When it's called it sets stdout to be in line
buffered mode. After that it used dlsym() to find the next
shared library in the list that has a function named "getenv". I
don't do any error checking since I know it exists. I then call it
and return whatever it returns.
I compiled the code with the following
cc -o linebuffered_stdout.so -fpic -shared linebuffered_stdout.c -ldl -lcThe result is a shared library named linebuffered_stdout.so. To see it in action I send mol2nam two lines of input. The first is "C" and the second is "CC" but there's a 4 second delay between them. Note the -u option to Python to make sure it doesn't buffer. I pipe the output side of mol2nam to cat -u. The pipe is to make mol2nam be in block buffer mode and the -u option to cat tells it to be unbuffered.
[~/src]$ python -u -c 'import time;print "C";time.sleep(4);print "CC"' | \ ? $OE_DIR/bin/mol2nam - | cat -u mol2nam v1.0 Structure to Name Conversion OpenEye Scientific Software, November 2003 methane ethane [~/src]$There is an obvious delay then both compound names are printed with no delay between them. This is the expected behaviour for block buffered mode.
Here's the version of the same test case using LD_PRELOAD to insert this shim that forces stdout to be line buffered.
[~/src]$ python -u -c 'import time;print "C";time.sleep(4);print "CC"' | \ ? env LD_PRELOAD=./linebuffered_stdout.so $OE_DIR/bin/mol2nam - | cat -u mol2nam v1.0 Structure to Name Conversion OpenEye Scientific Software, November 2003 methane ethane [~/src]If you watch it happen you'll see that "methane" is printed out then there's a pause of a few seconds before "ethane" is printed out. It works!
The different operating systems use different ways to affect the run-time loader. My primary machine is a Mac and after some 6 hours of searching, testing, and reading documentation and source still I couldn't figure out the equivalent to dlsym(RTLD_NEXT,...). Please let me know how to implement this sort of shim under that OS. (If I really needed something to work I would hard-code my shim getenv() to return the expected values and not worry about calling the original function.)
To make this work with the smi2name wrapper code I wrote a small shell script to set LD_PRELOAD correctly before starting mol2nam.
#!/bin/sh LD_PRELOAD=./linebuffered_stdout.so $OE_DIR/bin/mol2nam -I then changed the default MOL2NAM setting in the Python file to point to this wrapper shell script and ran the self test. After a few tens of seconds I was worried. Was the code stuck somewhere? Quit the program and add some print statements.
The problem was the segmentation fault test with the very large SMILES string. It didn't cause a segmentation fault under Linux. I made the string some 16 times larger and it still didn't crash. So I commented out that test and reran the test suite. Everything passed.
Andrew Dalke is an independent consultant focusing on software development for computational chemistry and biology. Need contract programming, help, or training? Contact me
Copyright © 2001-2020 Andrew Dalke Scientific AB