Incremental compilation

Incremental compilation

[Marc Nieper-Wißkirchen]

Hi!

I need to compile a number of functions incrementally. Eventually, I
would like to compile all the functions to a dynamic library file.

Can this be achieved with libgccjit without recompiling the functions?

-- Marc

Re: Incremental compilation

[Basile Starynkevitch]

On 12/29/18 4:31 PM, Marc Nieper-Wißkirchen wrote:
> Hi!
I am guessing that you use libgccjit on Linux.
>
> I need to compile a number of functions incrementally. Eventually, I
> would like to compile all the functions to a dynamic library file.
Notice that, if you are compiling each function incrementally, it has to 
be some exported function (GCC_JIT_FUNCTION_EXPORTED), otherwise you 
won't see it between your different compilations.
>
> Can this be achieved with libgccjit without recompiling the functions?

Probably not, and I don't understand why you need that in practice.

The common wisdom would be to compile each function separately (perhaps 
into its own shared object on Linux).

Be aware that it is perfectly acceptable to have a big lot (e.g. many 
hundred thousands) of shared objects on Linux.

My manydl.c program (which does not use libgccjit) on 
https://github.com/bstarynk/misc-basile/blob/master/manydl.c is 
generating some C code, compiling it as a plugin, and dlopen-ing it. It  
can repeat that a lot of times.

The only caveat when compiling small functions separately is that you 
have to spend one (or a few) pages (of 4Kbytes) on every compilation, 
since the result of a compilation fits in a separate page (e.g. in 
different *.so files). So if the function is tiny and takes only a 
hundred byte, you lose some virtual address space. In practice, that is 
no much a problem (at least on x86-64 under Linux).

My guess is that you are sort-of implementing some Read Eval Print Loop 
and JIT-compiling at every interaction. In practice, that works very 
well (even when emitting C code instead of using libgccjit). I was able 
to do that in my old GCC MELT project (which generated a C code, 
compiled it into a plugin, at every REPL interaction).

Happy new year to everyone.

-- 

Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France

Re: Incremental compilation

[Marc Nieper-Wißkirchen]

Am So., 30. Dez. 2018 um 09:46 Uhr schrieb Basile Starynkevitch
<basile@starynkevitch.net>:
>
>
> On 12/29/18 4:31 PM, Marc Nieper-Wißkirchen wrote:
> > Hi!
> I am guessing that you use libgccjit on Linux.
> >
> > I need to compile a number of functions incrementally. Eventually, I
> > would like to compile all the functions to a dynamic library file.
> Notice that, if you are compiling each function incrementally, it has to
> be some exported function (GCC_JIT_FUNCTION_EXPORTED), otherwise you
> won't see it between your different compilations.
> >
> > Can this be achieved with libgccjit without recompiling the functions?
>
> Probably not, and I don't understand why you need that in practice.

My use case is compiling a Scheme library. It contains macro
definitions whose transformation procedures have to be compiled at
first to expand subsequent parts of the library body. Furthermore, the
compiled transformation procedures of global macro defintions have to
appear in the final dynamic library as well.

>
> The common wisdom would be to compile each function separately (perhaps
> into its own shared object on Linux).
>
> Be aware that it is perfectly acceptable to have a big lot (e.g. many
> hundred thousands) of shared objects on Linux.

That's good to know!

So what I probably need is a kind of linker for libgccjit that
combines many of the compiled shared objects into one final module
that is then written to the disk (much like what LLVM's linker class
http://llvm.org/doxygen/classllvm_1_1Linker.html seems to achieve).

Another problem I have is that the resulting shared object has to note
its dependencies. However, libgccjit's documentation currently says:
"There is currently no support for specifying other libraries to link
against."

Is there a workaround or an ETA for this feature?

Thanks a lot!

-- Marc

P.S.: Where does libgccjit look for functions of kind
GCC_JIT_FUNCTION_IMPORTED? In the main executable and all dlopen'ed
libraries?

> My manydl.c program (which does not use libgccjit) on
> https://github.com/bstarynk/misc-basile/blob/master/manydl.c is
> generating some C code, compiling it as a plugin, and dlopen-ing it. It
> can repeat that a lot of times.
>
> The only caveat when compiling small functions separately is that you
> have to spend one (or a few) pages (of 4Kbytes) on every compilation,
> since the result of a compilation fits in a separate page (e.g. in
> different *.so files). So if the function is tiny and takes only a
> hundred byte, you lose some virtual address space. In practice, that is
> no much a problem (at least on x86-64 under Linux).
>
> My guess is that you are sort-of implementing some Read Eval Print Loop
> and JIT-compiling at every interaction. In practice, that works very
> well (even when emitting C code instead of using libgccjit). I was able
> to do that in my old GCC MELT project (which generated a C code,
> compiled it into a plugin, at every REPL interaction).
>
> Happy new year to everyone.
>
> --
>
> Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
> opinions are mine only - les opinions sont seulement miennes
> Bourg La Reine, France
>


-- 
Prof. Dr. Marc Nieper-Wißkirchen

Universität Augsburg
Institut für Mathematik
Universitätsstraße 14
86159 Augsburg

Tel: 0821/598-2146
Fax: 0821/598-2090

E-Mail: marc.nieper-wisskirchen@math.uni-augsburg.de
Web: www.math.uni-augsburg.de/alg/mitarbeiter/mnieper/

Re: Incremental compilation

[Basile Starynkevitch]

On 12/30/18 12:54 PM, Marc Nieper-Wißkirchen wrote:
> Am So., 30. Dez. 2018 um 09:46 Uhr schrieb Basile Starynkevitch
> <basile@starynkevitch.net>:
> My use case is compiling a Scheme library. It contains macro
> definitions whose transformation procedures have to be compiled at
> first to expand subsequent parts of the library body. Furthermore, the
> compiled transformation procedures of global macro defintions have to
> appear in the final dynamic library as well.
>
>> The common wisdom would be to compile each function separately (perhaps
>> into its own shared object on Linux).
>>
>> Be aware that it is perfectly acceptable to have a big lot (e.g. many
>> hundred thousands) of shared objects on Linux.
> That's good to know!
>
> So what I probably need is a kind of linker for libgccjit that
> combines many of the compiled shared objects into one final module
>
No you don't need that. You can keep the many compiled shared objects, 
and it would be actually simpler. You should not need to combine all 
these in a single module (except for optimization purposes, and for that 
special case, which in my view is not useful, you would recompile all 
your functions again).

In practice, since you are implementing a Scheme, you are dealing with 
closures, not plain C or GCCJIT functions. A closure contain a pointer 
to a function (and closed values).

Queinnec's book Lisp In Small Pieces should be highly relevant to you.

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France

Re: Incremental compilation

[Basile Starynkevitch]

On 12/30/18 12:54 PM, Marc Nieper-Wißkirchen wrote:

> My use case is compiling a Scheme library. It contains macro
> definitions whose transformation procedures have to be compiled at
> first to expand subsequent parts of the library body. Furthermore, the
> compiled transformation procedures of global macro defintions have to
> appear in the final dynamic library as well.

I am quite interested in your work, since I did (in the past) GCC MELT 
http://starynkevitch.net/Basile/gcc-melt/ (which now is a dead project; 
it was a Lisp dialect compiled to C or C++ for GCC extensions); I am 
working on http://github.com/bstarynk/bismon right now, and you might be 
interested in reading 
http://starynkevitch.net/Basile/bismon-chariot-doc.pdf

Is your project some open-source stuff? Could you give us the URL of its 
source code?

In MELT, I also had macros similar to yours. So I believe I understand 
your work quite well (since it is similar to what I did in GCC MELT, 
which was a Lisp -but not Scheme- dialect with macros). In practice, you 
still can compile every function in a separate *.so plugin.

You could group several (unrelated) functions together by JIT compiling 
them at once, but that is an optimization.

You could, once all macros have been compiled, regenerate a single 
"shared object" for all your library, but that is an optimization. The 
main reason to do so would be to avoid wasting virtual address space by 
having many thousands of shared objects. I am not sure it is worthwhile.

Since you are implementing a Scheme, all your functions are in fact used 
in closures (which contains a function pointer). So updating all the 
required closures (to put there a function pointer to a newer code) 
might be quite easy in practice.

The potential issue is to "garbage collect" useless code. In your case, 
that would mean doing dlclose in your GC. Do you want to do that?

Since you can in practice have many hundred thousands of useless 
generated plugins, you might decide to never call dlclose (and accept 
the "code segment leak" that follows).

Again, I really am interested by your work. If you have any additional 
references (draft reports, source code, etc...) please share them with 
us if possible.


Cheers

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France

Re: Incremental compilation

[Marc Nieper-Wißkirchen]

Am So., 30. Dez. 2018 um 13:28 Uhr schrieb Basile Starynkevitch
<basile@starynkevitch.net>:
>
>
> On 12/30/18 12:54 PM, Marc Nieper-Wißkirchen wrote:
>
> > My use case is compiling a Scheme library. It contains macro
> > definitions whose transformation procedures have to be compiled at
> > first to expand subsequent parts of the library body. Furthermore, the
> > compiled transformation procedures of global macro defintions have to
> > appear in the final dynamic library as well.
>
> I am quite interested in your work, since I did (in the past) GCC MELT
> http://starynkevitch.net/Basile/gcc-melt/ (which now is a dead project;
> it was a Lisp dialect compiled to C or C++ for GCC extensions); I am
> working on http://github.com/bstarynk/bismon right now, and you might be
> interested in reading
> http://starynkevitch.net/Basile/bismon-chariot-doc.pdf
>
> Is your project some open-source stuff? Could you give us the URL of its
> source code?

At the moment, most parts of the project is just in my head. I have
written some code to be able to do some experiments but nothing is
finished yet. As soon as I have something presentable, I'd love to
share it with you. It will be GPL'd code.

The goal of the project is to have a reasonably fast Scheme
implementation that I can use for experiments with language extension.

> In MELT, I also had macros similar to yours. So I believe I understand
> your work quite well (since it is similar to what I did in GCC MELT,
> which was a Lisp -but not Scheme- dialect with macros). In practice, you
> still can compile every function in a separate *.so plugin.
>
> You could group several (unrelated) functions together by JIT compiling
> them at once, but that is an optimization.
>
> You could, once all macros have been compiled, regenerate a single
> "shared object" for all your library, but that is an optimization. The
> main reason to do so would be to avoid wasting virtual address space by
> having many thousands of shared objects. I am not sure it is worthwhile.

The reason why I would like to have a single shared object is for the
convenience of the user. The compiler should produce a single shared
object from a Scheme library definition which can then later be
visited/invoked in lieu of loading the Scheme source code of the
library.

> Since you are implementing a Scheme, all your functions are in fact used
> in closures (which contains a function pointer). So updating all the
> required closures (to put there a function pointer to a newer code)
> might be quite easy in practice.
>
> The potential issue is to "garbage collect" useless code. In your case,
> that would mean doing dlclose in your GC. Do you want to do that?

There will be two sources of (longer lived) compiled code: Libraries
that have been loaded and functions generated by `eval'. The former
will never be garbage collected. The latter will. `Eval' would
(internally) return closures created that should be disposable by
calling 'gcc_jit_result_release`.

> Since you can in practice have many hundred thousands of useless
> generated plugins, you might decide to never call dlclose (and accept
> the "code segment leak" that follows).
>
> Again, I really am interested by your work. If you have any additional
> references (draft reports, source code, etc...) please share them with
> us if possible.

I really should write up my concept. It isn't going to be finished
this year, however.

Best,

Marc

>
>
> Cheers
>
> --
> Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
> opinions are mine only - les opinions sont seulement miennes
> Bourg La Reine, France
>


-- 
Prof. Dr. Marc Nieper-Wißkirchen

Universität Augsburg
Institut für Mathematik
Universitätsstraße 14
86159 Augsburg

Tel: 0821/598-2146
Fax: 0821/598-2090

E-Mail: marc.nieper-wisskirchen@math.uni-augsburg.de
Web: www.math.uni-augsburg.de/alg/mitarbeiter/mnieper/

Re: Incremental compilation

[Basile Starynkevitch]

On 12/30/18 4:04 PM, Marc Nieper-Wißkirchen wrote:
> Am So., 30. Dez. 2018 um 13:28 Uhr schrieb Basile Starynkevitch
> <basile@starynkevitch.net>:
>
>
> You could, once all macros have been compiled, regenerate a single
> "shared object" for all your library, but that is an optimization. The
> main reason to do so would be to avoid wasting virtual address space by
> having many thousands of shared objects. I am not sure it is worthwhile.
> The reason why I would like to have a single shared object is for the
> convenience of the user. The compiler should produce a single shared
> object from a Scheme library definition which can then later be
> visited/invoked in lieu of loading the Scheme source code of the
> library.

Then you could compile twice the code.

Alternatively, you could compile each function once into a .pic.o
position-independent code object file. When doing the incremental compilation, you also need to convert each of these into its own .so file
(and you would fork a ld or gcc process for that; it should run quickly).

BTW, since libgccjit is really a wrapper above GCC, you could generate once the position-indenpendent code object files
(so usinggcc_jit_context_add_command_line_option to ask for -fPIC then using 
gcc_jit_context_compile_to_file) of each function. Then you would fork a 
gcc or ld process to link all these *.pic.o into a single .so; this 
would be reasonably quick. Take advantage that libgccjit is just a 
front-end to GCC. It fakes to be a JIT compiler, but it is not really 
one. Its main advantage is to avoid requiring you to emit C code in 
files. Cheers.
-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France

Re: Incremental compilation

[Marc Nieper-Wißkirchen]

Am So., 30. Dez. 2018 um 16:53 Uhr schrieb Basile Starynkevitch
<basile@starynkevitch.net>:
>
>
> On 12/30/18 4:04 PM, Marc Nieper-Wißkirchen wrote:
> > Am So., 30. Dez. 2018 um 13:28 Uhr schrieb Basile Starynkevitch
> > <basile@starynkevitch.net>:
> >
> >
> > You could, once all macros have been compiled, regenerate a single
> > "shared object" for all your library, but that is an optimization. The
> > main reason to do so would be to avoid wasting virtual address space by
> > having many thousands of shared objects. I am not sure it is worthwhile.
> > The reason why I would like to have a single shared object is for the
> > convenience of the user. The compiler should produce a single shared
> > object from a Scheme library definition which can then later be
> > visited/invoked in lieu of loading the Scheme source code of the
> > library.
>
> Then you could compile twice the code.
>
> Alternatively, you could compile each function once into a .pic.o
> position-independent code object file. When doing the incremental compilation, you also need to convert each of these into its own .so file
> (and you would fork a ld or gcc process for that; it should run quickly).
>
> BTW, since libgccjit is really a wrapper above GCC, you could generate once the position-indenpendent code object files
> (so usinggcc_jit_context_add_command_line_option to ask for -fPIC then using
> gcc_jit_context_compile_to_file) of each function. Then you would fork a
> gcc or ld process to link all these *.pic.o into a single .so; this
> would be reasonably quick. Take advantage that libgccjit is just a
> front-end to GCC. It fakes to be a JIT compiler, but it is not really
> one. Its main advantage is to avoid requiring you to emit C code in
> files. Cheers.

Another advantage of libgccjit (vs invoking GCC directly) is that it
abstracts away issues like locating the GCC binary on the target
system, handling of temporary files, etc.

Maybe it makes sense if libgccjit exposes that part as an API to the
user as well.

-- Marc