Dynamic compilation using gcc

Dynamic compilation using gcc

[Klaus Grue]

Hi gcc-help,

I have a three questions about dynamic compilation using gcc which I post 
to gcc-help since I have been unable to find much on the issue using FAQs 
and Google. I hope someone can provide answers or pointers to information.

Background: I need a function

   void *source2dl(const char *source,int flag);

which, given a C source text, compiles the source into a dynamic library 
and returns a handle to that library. The signature of source2dl() above 
is identical to that of dlopen(), but source2dl() takes a C source as 
input where dlopen() takes the pathname of a shared object.

I include a working but in some respects inconvenient implementation of 
source2dl() below. My questions come after the implementation of 
source2dl().

---------------------------------------------------------------------

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/wait.h>
#include <dlfcn.h>

//Die with last word from perror
void pdie(const char *msg){perror(msg);exit(0);}

//Die with last word from dlerror
void dldie(const char *msg){printf("%s: %s\n",msg,dlerror());exit(0);}

//Convert source text to dynamic library
void *source2dl(const char *source,int flag){
   int fd[2];pid_t pid;int status;void *dl;
//Create pipe for piping source to gcc
   if(pipe(fd)<0) pdie("pipe()");
//Fork a gcc, pipe source to it, wait for gcc to exit
   pid=fork();
   if(pid<0) pdie("fork()");
   if(pid==0){
//  Child process: close write end of pipe
     close(fd[1]);
//  Send pipe input to stdin
     close(STDIN_FILENO);
     dup2(fd[0],STDIN_FILENO);
//  Invoke gcc with the following arguments:
//  -x c               Tell gcc that we provide C source
//  -pipe              Use pipes between gcc stages
//  -fPIC -shared      Make shared object
//  -Wl,-soname,dl.so  Pass suitable options to linker
//  -o dl.so           Write shared object to dl.so
     execl("/usr/bin/gcc","gcc","-x","c","-pipe","-","-fPIC","-shared",
       "-Wl,-soname,dl.so","-o","dl.so",(char *)NULL);
//  Complain if execl() returns
     pdie("execl()");}
//Parent process: close read end of pipe
   close(fd[0]);
//Pipe source to gcc
   write(fd[1],source,strlen(source));
   close(fd[1]);
//Wait for gcc to exit
   if(wait(&status)<0) pdie("wait()");
//Load output from gcc and return it
   dl=dlopen("./dl.so",flag);
   if(!dl) dldie("dlopen()");
   return dl;}

//Main program for testing source2dl()
int main(int argc,char *argv[]){
//Set prog to
//#include <stdio.h>
//void fct(){printf("Hello world\n");}
   char *prog="#include <stdio.h>\nvoid fct(){printf(\"Hello 
world\\n\");}\n";
//Convert prog to dl and extract fct
   void (*fct)()=dlsym(source2dl(prog,RTLD_NOW),"fct");
   if(!fct) dldie("dlsym()");
//Invoke fct
   (*fct)();
   return 0;}

---------------------------------------------------------------------

The program above prints "Hello world" to stdout as intended. However, 
here are my questions:

Question 1. The dlopen(), dlclose(), dlsym(), and dlerror() functions from 
libdl provide a library interface to the dynamic loader. Does there exist 
a similar library interface to gcc so that one can avoid the 
pipe-fork-execve-wait overhead of running gcc in a separate process?

Question 2. The implementation of source2dl() above uses a temporary file 
named dl.so for the shared object produced by gcc and read by dlopen. It 
would be convenient if communication from gcc to dlopen could go via a 
pipe. Does anyone know a way to tell gcc to write the shared object to 
stdout?

Question 3. Following up on Question 2, does anyone know a way to tell 
glopen to read the shared object from stdin?

I hope someone listening at gcc-help can help on these questions or give 
pointers to information or give suggestions for alternative ways of doing 
dynamic compilation.

Cheers,
Klaus

Re: Dynamic compilation using gcc

[Brian Dessent]

Klaus Grue wrote:

> Question 1. The dlopen(), dlclose(), dlsym(), and dlerror() functions from
> libdl provide a library interface to the dynamic loader. Does there exist
> a similar library interface to gcc so that one can avoid the
> pipe-fork-execve-wait overhead of running gcc in a separate process?

No, nothing like that exists.  The closest thing is probably the
incremental compiler project, currently under development on a branch. 
The goal is to turn gcc into a long-running daemon/server process to
avoid the startup cost of having to invoke gcc for each object, and to
be able to re-use the results of parsing common header files,
potentially with multiple threads.  But there's no exported C API for
this as far as I know, it would still continue to interface with the
normal gcc command line invocation.

> Question 2. The implementation of source2dl() above uses a temporary file
> named dl.so for the shared object produced by gcc and read by dlopen. It
> would be convenient if communication from gcc to dlopen could go via a
> pipe. Does anyone know a way to tell gcc to write the shared object to
> stdout?

Well first, gcc doesn't write to the shared object file.  The only thing
gcc ever writes is assembly to an .s file, or in this case since you
used -pipe, that goes over a pipe to the assembler which writes to an .o
file, which is a temporary file somewhere in $TMP since you aren't using
-c.  Then gcc invokes collect2, which in turn invokes ld, which is what
actually reads the .o file and writes the .so file.  ld cannot write to
a pipe, it must write to a file.  Even if it could, you still have a
temporary file that is created and deleted in the form of the .o, so I
don't think it's ever going to be possible to do this without creating a
file.  The toolchain just wasn't designed to be used like that.

> Question 3. Following up on Question 2, does anyone know a way to tell
> glopen to read the shared object from stdin?

dlopen has to mmap() the file into memory to execute it, so there has to
be an underlying file somewhere.  I don't think you can get around
that.  You could of course create the file in a tmpfs filesystem so that
it's all in memory, and unlink it after it has been dlopened so that
it's not accessible through any directory entry.  If you do this then it
will automatically be reclaimed by the filesystem whenever the last open
handle to it closes, i.e. the process that has it mapped terminates.

> I hope someone listening at gcc-help can help on these questions or give
> pointers to information or give suggestions for alternative ways of doing
> dynamic compilation.

I don't see what's so bad about using files.  unlinking an open file is
the standard technique when you have a temporary file that you want to
not be accessible to other processes and that you want to be
automatically cleaned up by the operating system when it's no longer in
use.

Brian

Re: Dynamic compilation using gcc

[Klaus Grue]

Dear Brian,

Thanks a lot for your answers and background information.

> No, nothing like [a C API] exists...
> dlopen has to mmap() the file...
> You could of course create the file in a tmpfs filesystem...
> I don't see what's so bad about using files.
> unlinking an open file is the standard technique...

I agree. Using files is not that bad. In fact, with the information you 
gave it seems to be the way to go. Thanks a lot. Your answers will save me 
a lot of time searching for something which does not exist.

Cheers,
Klaus