* Incremental compilation @ 2018-01-01 0:00 Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch 0 siblings, 1 reply; 8+ messages in thread From: Marc Nieper-Wißkirchen @ 2018-01-01 0:00 UTC (permalink / raw) To: jit Hi! I need to compile a number of functions incrementally. Eventually, I would like to compile all the functions to a dynamic library file. Can this be achieved with libgccjit without recompiling the functions? -- Marc ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Incremental compilation 2018-01-01 0:00 Incremental compilation Marc Nieper-Wißkirchen @ 2018-01-01 0:00 ` Basile Starynkevitch 2018-01-01 0:00 ` Marc Nieper-Wißkirchen 0 siblings, 1 reply; 8+ messages in thread From: Basile Starynkevitch @ 2018-01-01 0:00 UTC (permalink / raw) To: Marc Nieper-Wißkirchen, jit On 12/29/18 4:31 PM, Marc Nieper-WiÃkirchen wrote: > Hi! I am guessing that you use libgccjit on Linux. > > I need to compile a number of functions incrementally. Eventually, I > would like to compile all the functions to a dynamic library file. Notice that, if you are compiling each function incrementally, it has to be some exported function (GCC_JIT_FUNCTION_EXPORTED), otherwise you won't see it between your different compilations. > > Can this be achieved with libgccjit without recompiling the functions? Probably not, and I don't understand why you need that in practice. The common wisdom would be to compile each function separately (perhaps into its own shared object on Linux). Be aware that it is perfectly acceptable to have a big lot (e.g. many hundred thousands) of shared objects on Linux. My manydl.c program (which does not use libgccjit) on https://github.com/bstarynk/misc-basile/blob/master/manydl.c is generating some C code, compiling it as a plugin, and dlopen-ing it. It can repeat that a lot of times. The only caveat when compiling small functions separately is that you have to spend one (or a few) pages (of 4Kbytes) on every compilation, since the result of a compilation fits in a separate page (e.g. in different *.so files). So if the function is tiny and takes only a hundred byte, you lose some virtual address space. In practice, that is no much a problem (at least on x86-64 under Linux). My guess is that you are sort-of implementing some Read Eval Print Loop and JIT-compiling at every interaction. In practice, that works very well (even when emitting C code instead of using libgccjit). I was able to do that in my old GCC MELT project (which generated a C code, compiled it into a plugin, at every REPL interaction). Happy new year to everyone. -- Basile STARYNKEVITCH == http://starynkevitch.net/Basile opinions are mine only - les opinions sont seulement miennes Bourg La Reine, France ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Incremental compilation 2018-01-01 0:00 ` Basile Starynkevitch @ 2018-01-01 0:00 ` Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch 2018-01-01 0:00 ` Basile Starynkevitch 0 siblings, 2 replies; 8+ messages in thread From: Marc Nieper-Wißkirchen @ 2018-01-01 0:00 UTC (permalink / raw) To: Basile Starynkevitch; +Cc: jit Am So., 30. Dez. 2018 um 09:46 Uhr schrieb Basile Starynkevitch <basile@starynkevitch.net>: > > > On 12/29/18 4:31 PM, Marc Nieper-Wißkirchen wrote: > > Hi! > I am guessing that you use libgccjit on Linux. > > > > I need to compile a number of functions incrementally. Eventually, I > > would like to compile all the functions to a dynamic library file. > Notice that, if you are compiling each function incrementally, it has to > be some exported function (GCC_JIT_FUNCTION_EXPORTED), otherwise you > won't see it between your different compilations. > > > > Can this be achieved with libgccjit without recompiling the functions? > > Probably not, and I don't understand why you need that in practice. My use case is compiling a Scheme library. It contains macro definitions whose transformation procedures have to be compiled at first to expand subsequent parts of the library body. Furthermore, the compiled transformation procedures of global macro defintions have to appear in the final dynamic library as well. > > The common wisdom would be to compile each function separately (perhaps > into its own shared object on Linux). > > Be aware that it is perfectly acceptable to have a big lot (e.g. many > hundred thousands) of shared objects on Linux. That's good to know! So what I probably need is a kind of linker for libgccjit that combines many of the compiled shared objects into one final module that is then written to the disk (much like what LLVM's linker class http://llvm.org/doxygen/classllvm_1_1Linker.html seems to achieve). Another problem I have is that the resulting shared object has to note its dependencies. However, libgccjit's documentation currently says: "There is currently no support for specifying other libraries to link against." Is there a workaround or an ETA for this feature? Thanks a lot! -- Marc P.S.: Where does libgccjit look for functions of kind GCC_JIT_FUNCTION_IMPORTED? In the main executable and all dlopen'ed libraries? > My manydl.c program (which does not use libgccjit) on > https://github.com/bstarynk/misc-basile/blob/master/manydl.c is > generating some C code, compiling it as a plugin, and dlopen-ing it. It > can repeat that a lot of times. > > The only caveat when compiling small functions separately is that you > have to spend one (or a few) pages (of 4Kbytes) on every compilation, > since the result of a compilation fits in a separate page (e.g. in > different *.so files). So if the function is tiny and takes only a > hundred byte, you lose some virtual address space. In practice, that is > no much a problem (at least on x86-64 under Linux). > > My guess is that you are sort-of implementing some Read Eval Print Loop > and JIT-compiling at every interaction. In practice, that works very > well (even when emitting C code instead of using libgccjit). I was able > to do that in my old GCC MELT project (which generated a C code, > compiled it into a plugin, at every REPL interaction). > > Happy new year to everyone. > > -- > > Basile STARYNKEVITCH == http://starynkevitch.net/Basile > opinions are mine only - les opinions sont seulement miennes > Bourg La Reine, France > -- Prof. Dr. Marc Nieper-Wißkirchen Universität Augsburg Institut für Mathematik Universitätsstraße 14 86159 Augsburg Tel: 0821/598-2146 Fax: 0821/598-2090 E-Mail: marc.nieper-wisskirchen@math.uni-augsburg.de Web: www.math.uni-augsburg.de/alg/mitarbeiter/mnieper/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Incremental compilation 2018-01-01 0:00 ` Marc Nieper-Wißkirchen @ 2018-01-01 0:00 ` Basile Starynkevitch 2018-01-01 0:00 ` Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch 1 sibling, 1 reply; 8+ messages in thread From: Basile Starynkevitch @ 2018-01-01 0:00 UTC (permalink / raw) To: Marc Nieper-Wißkirchen; +Cc: jit On 12/30/18 12:54 PM, Marc Nieper-WiÃkirchen wrote: > My use case is compiling a Scheme library. It contains macro > definitions whose transformation procedures have to be compiled at > first to expand subsequent parts of the library body. Furthermore, the > compiled transformation procedures of global macro defintions have to > appear in the final dynamic library as well. I am quite interested in your work, since I did (in the past) GCC MELT http://starynkevitch.net/Basile/gcc-melt/ (which now is a dead project; it was a Lisp dialect compiled to C or C++ for GCC extensions); I am working on http://github.com/bstarynk/bismon right now, and you might be interested in reading http://starynkevitch.net/Basile/bismon-chariot-doc.pdf Is your project some open-source stuff? Could you give us the URL of its source code? In MELT, I also had macros similar to yours. So I believe I understand your work quite well (since it is similar to what I did in GCC MELT, which was a Lisp -but not Scheme- dialect with macros). In practice, you still can compile every function in a separate *.so plugin. You could group several (unrelated) functions together by JIT compiling them at once, but that is an optimization. You could, once all macros have been compiled, regenerate a single "shared object" for all your library, but that is an optimization. The main reason to do so would be to avoid wasting virtual address space by having many thousands of shared objects. I am not sure it is worthwhile. Since you are implementing a Scheme, all your functions are in fact used in closures (which contains a function pointer). So updating all the required closures (to put there a function pointer to a newer code) might be quite easy in practice. The potential issue is to "garbage collect" useless code. In your case, that would mean doing dlclose in your GC. Do you want to do that? Since you can in practice have many hundred thousands of useless generated plugins, you might decide to never call dlclose (and accept the "code segment leak" that follows). Again, I really am interested by your work. If you have any additional references (draft reports, source code, etc...) please share them with us if possible. Cheers -- Basile STARYNKEVITCH == http://starynkevitch.net/Basile opinions are mine only - les opinions sont seulement miennes Bourg La Reine, France ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Incremental compilation 2018-01-01 0:00 ` Basile Starynkevitch @ 2018-01-01 0:00 ` Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch 0 siblings, 1 reply; 8+ messages in thread From: Marc Nieper-Wißkirchen @ 2018-01-01 0:00 UTC (permalink / raw) To: Basile Starynkevitch; +Cc: jit Am So., 30. Dez. 2018 um 13:28 Uhr schrieb Basile Starynkevitch <basile@starynkevitch.net>: > > > On 12/30/18 12:54 PM, Marc Nieper-Wißkirchen wrote: > > > My use case is compiling a Scheme library. It contains macro > > definitions whose transformation procedures have to be compiled at > > first to expand subsequent parts of the library body. Furthermore, the > > compiled transformation procedures of global macro defintions have to > > appear in the final dynamic library as well. > > I am quite interested in your work, since I did (in the past) GCC MELT > http://starynkevitch.net/Basile/gcc-melt/ (which now is a dead project; > it was a Lisp dialect compiled to C or C++ for GCC extensions); I am > working on http://github.com/bstarynk/bismon right now, and you might be > interested in reading > http://starynkevitch.net/Basile/bismon-chariot-doc.pdf > > Is your project some open-source stuff? Could you give us the URL of its > source code? At the moment, most parts of the project is just in my head. I have written some code to be able to do some experiments but nothing is finished yet. As soon as I have something presentable, I'd love to share it with you. It will be GPL'd code. The goal of the project is to have a reasonably fast Scheme implementation that I can use for experiments with language extension. > In MELT, I also had macros similar to yours. So I believe I understand > your work quite well (since it is similar to what I did in GCC MELT, > which was a Lisp -but not Scheme- dialect with macros). In practice, you > still can compile every function in a separate *.so plugin. > > You could group several (unrelated) functions together by JIT compiling > them at once, but that is an optimization. > > You could, once all macros have been compiled, regenerate a single > "shared object" for all your library, but that is an optimization. The > main reason to do so would be to avoid wasting virtual address space by > having many thousands of shared objects. I am not sure it is worthwhile. The reason why I would like to have a single shared object is for the convenience of the user. The compiler should produce a single shared object from a Scheme library definition which can then later be visited/invoked in lieu of loading the Scheme source code of the library. > Since you are implementing a Scheme, all your functions are in fact used > in closures (which contains a function pointer). So updating all the > required closures (to put there a function pointer to a newer code) > might be quite easy in practice. > > The potential issue is to "garbage collect" useless code. In your case, > that would mean doing dlclose in your GC. Do you want to do that? There will be two sources of (longer lived) compiled code: Libraries that have been loaded and functions generated by `eval'. The former will never be garbage collected. The latter will. `Eval' would (internally) return closures created that should be disposable by calling 'gcc_jit_result_release`. > Since you can in practice have many hundred thousands of useless > generated plugins, you might decide to never call dlclose (and accept > the "code segment leak" that follows). > > Again, I really am interested by your work. If you have any additional > references (draft reports, source code, etc...) please share them with > us if possible. I really should write up my concept. It isn't going to be finished this year, however. Best, Marc > > > Cheers > > -- > Basile STARYNKEVITCH == http://starynkevitch.net/Basile > opinions are mine only - les opinions sont seulement miennes > Bourg La Reine, France > -- Prof. Dr. Marc Nieper-Wißkirchen Universität Augsburg Institut für Mathematik Universitätsstraße 14 86159 Augsburg Tel: 0821/598-2146 Fax: 0821/598-2090 E-Mail: marc.nieper-wisskirchen@math.uni-augsburg.de Web: www.math.uni-augsburg.de/alg/mitarbeiter/mnieper/ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Incremental compilation 2018-01-01 0:00 ` Marc Nieper-Wißkirchen @ 2018-01-01 0:00 ` Basile Starynkevitch 2019-01-01 0:00 ` Marc Nieper-Wißkirchen 0 siblings, 1 reply; 8+ messages in thread From: Basile Starynkevitch @ 2018-01-01 0:00 UTC (permalink / raw) To: Marc Nieper-Wißkirchen; +Cc: jit On 12/30/18 4:04 PM, Marc Nieper-WiÃkirchen wrote: > Am So., 30. Dez. 2018 um 13:28 Uhr schrieb Basile Starynkevitch > <basile@starynkevitch.net>: > > > You could, once all macros have been compiled, regenerate a single > "shared object" for all your library, but that is an optimization. The > main reason to do so would be to avoid wasting virtual address space by > having many thousands of shared objects. I am not sure it is worthwhile. > The reason why I would like to have a single shared object is for the > convenience of the user. The compiler should produce a single shared > object from a Scheme library definition which can then later be > visited/invoked in lieu of loading the Scheme source code of the > library. Then you could compile twice the code. Alternatively, you could compile each function once into a .pic.o position-independent code object file. When doing the incremental compilation, you also need to convert each of these into its own .so file (and you would fork a ld or gcc process for that; it should run quickly). BTW, since libgccjit is really a wrapper above GCC, you could generate once the position-indenpendent code object files (so usinggcc_jit_context_add_command_line_option to ask for -fPIC then using gcc_jit_context_compile_to_file) of each function. Then you would fork a gcc or ld process to link all these *.pic.o into a single .so; this would be reasonably quick. Take advantage that libgccjit is just a front-end to GCC. It fakes to be a JIT compiler, but it is not really one. Its main advantage is to avoid requiring you to emit C code in files. Cheers. -- Basile STARYNKEVITCH == http://starynkevitch.net/Basile opinions are mine only - les opinions sont seulement miennes Bourg La Reine, France ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Incremental compilation 2018-01-01 0:00 ` Basile Starynkevitch @ 2019-01-01 0:00 ` Marc Nieper-Wißkirchen 0 siblings, 0 replies; 8+ messages in thread From: Marc Nieper-Wißkirchen @ 2019-01-01 0:00 UTC (permalink / raw) To: Basile Starynkevitch; +Cc: jit Am So., 30. Dez. 2018 um 16:53 Uhr schrieb Basile Starynkevitch <basile@starynkevitch.net>: > > > On 12/30/18 4:04 PM, Marc Nieper-Wißkirchen wrote: > > Am So., 30. Dez. 2018 um 13:28 Uhr schrieb Basile Starynkevitch > > <basile@starynkevitch.net>: > > > > > > You could, once all macros have been compiled, regenerate a single > > "shared object" for all your library, but that is an optimization. The > > main reason to do so would be to avoid wasting virtual address space by > > having many thousands of shared objects. I am not sure it is worthwhile. > > The reason why I would like to have a single shared object is for the > > convenience of the user. The compiler should produce a single shared > > object from a Scheme library definition which can then later be > > visited/invoked in lieu of loading the Scheme source code of the > > library. > > Then you could compile twice the code. > > Alternatively, you could compile each function once into a .pic.o > position-independent code object file. When doing the incremental compilation, you also need to convert each of these into its own .so file > (and you would fork a ld or gcc process for that; it should run quickly). > > BTW, since libgccjit is really a wrapper above GCC, you could generate once the position-indenpendent code object files > (so usinggcc_jit_context_add_command_line_option to ask for -fPIC then using > gcc_jit_context_compile_to_file) of each function. Then you would fork a > gcc or ld process to link all these *.pic.o into a single .so; this > would be reasonably quick. Take advantage that libgccjit is just a > front-end to GCC. It fakes to be a JIT compiler, but it is not really > one. Its main advantage is to avoid requiring you to emit C code in > files. Cheers. Another advantage of libgccjit (vs invoking GCC directly) is that it abstracts away issues like locating the GCC binary on the target system, handling of temporary files, etc. Maybe it makes sense if libgccjit exposes that part as an API to the user as well. -- Marc ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Incremental compilation 2018-01-01 0:00 ` Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch @ 2018-01-01 0:00 ` Basile Starynkevitch 1 sibling, 0 replies; 8+ messages in thread From: Basile Starynkevitch @ 2018-01-01 0:00 UTC (permalink / raw) To: Marc Nieper-Wißkirchen; +Cc: jit On 12/30/18 12:54 PM, Marc Nieper-WiÃkirchen wrote: > Am So., 30. Dez. 2018 um 09:46 Uhr schrieb Basile Starynkevitch > <basile@starynkevitch.net>: > My use case is compiling a Scheme library. It contains macro > definitions whose transformation procedures have to be compiled at > first to expand subsequent parts of the library body. Furthermore, the > compiled transformation procedures of global macro defintions have to > appear in the final dynamic library as well. > >> The common wisdom would be to compile each function separately (perhaps >> into its own shared object on Linux). >> >> Be aware that it is perfectly acceptable to have a big lot (e.g. many >> hundred thousands) of shared objects on Linux. > That's good to know! > > So what I probably need is a kind of linker for libgccjit that > combines many of the compiled shared objects into one final module > No you don't need that. You can keep the many compiled shared objects, and it would be actually simpler. You should not need to combine all these in a single module (except for optimization purposes, and for that special case, which in my view is not useful, you would recompile all your functions again). In practice, since you are implementing a Scheme, you are dealing with closures, not plain C or GCCJIT functions. A closure contain a pointer to a function (and closed values). Queinnec's book Lisp In Small Pieces should be highly relevant to you. -- Basile STARYNKEVITCH == http://starynkevitch.net/Basile opinions are mine only - les opinions sont seulement miennes Bourg La Reine, France ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-01-01 12:08 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-01-01 0:00 Incremental compilation Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch 2018-01-01 0:00 ` Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch 2018-01-01 0:00 ` Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch 2019-01-01 0:00 ` Marc Nieper-Wißkirchen 2018-01-01 0:00 ` Basile Starynkevitch
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).