Re: about header file parsing

Re: about header file parsing

[akrl]

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Am So., 13. Jan. 2019 um 20:56 Uhr schrieb akrl <akrl@sdf.org>:
>
> [...]
>
>> Hi Marc,
>> there's no technical problem into including these into the patch.
>> But unless we have some special reason for that (like minimizing
>> LIBGCCJIT_ABI changes) I would rather prefer to have this changes as a
>> following patch.
>> But I guess is good to have Dave's opinion on all of these points.
>
> Indeed. What is the general policy with respect to the ABI version?
> One version bump per patch/feature or one version bump per release
> cycle?
>
> [...]
>
> Best,
>
> Marc
>

Okay I've submitted the patch for exposing gcc_jit_context_add_driver_option.

If Dave's agree on the idea of the modification I'll have a following
patch for exposing invoke_embedded_driver and invoke_external_driver
after the review process of the first is done.

Bests
   Andrea

-- 
akrl@sdf.org

Re: about header file parsing

[Marc Nieper-Wißkirchen]

Am Di., 8. Jan. 2019 um 22:16 Uhr schrieb Basile Starynkevitch
<basile@starynkevitch.net>:

[...]

> Still another thing could be to use LTO: you'll compile your C file with
> GCC using -flto, you'll do LIBGCCJIT things, and the final executable
> sould be compiled and linked with -flto.

Is using:

gcc_jit_context_add_command_line_option (ctxt, "-flto helper.o");

supported to have the object generated by libgccjit statically linked
with the object helper.o before libgccjit emits the final dynamic
library? (Let's assume that helper.o has been compiled with -flto.)

[...]

-- Marc

Re: about header file parsing

[David Malcolm]

On Wed, 2019-01-09 at 16:13 +0100, Basile Starynkevitch wrote:
> On 1/9/19 4:03 PM, akrl wrote:
> > Sorry I try to explain it better.
> > I was thinking to a gcc plugin (or potentially something else)
> > that 
> > given a C file produce the libgccjit code necessary to fill a
> > context
> > with what is in the source file.
> 
> That could not work in general, because C has macros (think of the
> `EOF` 
> macro constant in `<stdio.h>`) that have no equivalent in libgccjit. 

Although possibly not working "in general" I think it could work for
most important cases.   A macro in C typically can be handled as either
a client-side function invocation in libgccjit, or as a libgccjit
function.

For example, CPython has various reference-counting macros:

 #define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)

and in my "coconut" JIT-compiler for CPython I implement this as a
libgccjit function:

https://github.com/davidmalcolm/coconut/blob/master/coconut/compiler.py
#L501

(effectively treating it as a static inline function).

> And 
> in practice C macros are very important (and a lot of API for C are 
> using macros, see also macros such as WIFEXITED in wait(2) on some
> Linux 
> system).
> 
> We have to admit that C and libgccjit are different "languages" (even
> if 
> conceptually very close). And a C API is not exactly a set of
> "objects" 
> that libgccjit can handle.

They're not the same, but it is important that libgccjit be able to
interoperate well with C APIs.

Dave

Re: about header file parsing

[akrl]

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Am Fr., 18. Jan. 2019 um 21:10 Uhr schrieb akrl <akrl@sdf.org>:
>
> [...]
>
>> Okay I've submitted the patch for exposing gcc_jit_context_add_driver_option.
>
> That's great!
>
> Have you experimented with your patch and checked whether linking
> object statically with link-time optimization to the
> libgccjit-generated code works?

Seams to work for me.
I've verified I can inline a function defined in an external obj file
test.o.

This is what I had into the jitter code as conf:

gcc_jit_context_set_int_option(ctxt, GCC_JIT_INT_OPTION_OPTIMIZATION_LEVEL, 3);
gcc_jit_context_add_command_line_option(ctxt, "-flto");

gcc_jit_context_add_driver_option(ctxt, "-fuse-linker-plugin");
gcc_jit_context_add_driver_option(ctxt, "-flto");
gcc_jit_context_add_driver_option(ctxt, "test.o");

>> If Dave's agree on the idea of the modification I'll have a following
>> patch for exposing invoke_embedded_driver and invoke_external_driver
>> after the review process of the first is done.
>
> Very cool!
>
> [...]
>
> Thanks,
>
> Marc
>

Very welcome :)

Bests
   Andrea

-- 
akrl@sdf.org

Re: about header file parsing

[Basile Starynkevitch]

On 1/9/19 12:12 PM, akrl wrote:
>
> I think the idea of having something like a gcc plugin that once is parsed
> compiles the subset of supported C to the necessary libgccjit code would be
> useful to have for this cases but also educative for learning how to use the
> library.
>
> Would be this something that is upstreamable in some form?
>
I don't know of any GCC plugin within the GCC code base (except the test cases for plugins).

And I am not sure it would be that easy. If I understand well,
what you suggest would either fill some gcc_jit_context, or create it already filled with an API.

But, AFAIK, there is no current way to query the content of some gcc_jit_context. In other words, it
seems (but I could be wrong) that what is lacking is an API to e.g. *fetch* all the struct-ures and
all the functions inside a gcc_jit_context, and we don't have that yet.

Or perhaps, I have misundertood your idea.

My feeling was that the GCC plugin would fill whatever data structure is used by the application using libgccjit.
And that is application specific -so the GCC plugin has to be application specific too.

Cheers

PS. I could be very wrong, it is not clear in my head!

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France

Re: about header file parsing

[akrl]

Marc Nieper-Wißkirchen <marc.nieper@gmail.com> writes:

> How would you handle, for example, gmp's header file?
> https://gmplib.org/repo/gmp/file/default/gmp-h.in
>
> Almost all of the public API is behind #define's. So even if the
> gcc_jit_context is filled with the functions that are declared in the
> header, one still needs to know the ABI of GMP to know which functions
> to call.
>
> In order to make it work in the general case (if this is at all
> possible), I think one has to turn the C frontend of GCC into a
> library (like libgccjit has turned the GCC middle/backend into a
> library). This hypothetical library X has its own context K that is
> populated by expanding and parsing the relevant header files.
>
> In this context K, we can then ask for expansion of strings of C
> tokens (e.g. in expression or statement context). Library X then
> returns the expansion in term of, say, a C-specific GENERIC tree,
> which can then be converted into/wrapped into a gcc_jit_object.
>
> -- Marc

So if I undertand correctly: the problem you are raising about defines is
 about naming in the sense that to use an API under defines you'll
need to use the the preprocessed expanded names in libgccjit?

If this is the case I still think the tool would be useful.

Andrea

-- 
akrl@sdf.org

Re: about header file parsing

[Basile Starynkevitch]

On 1/8/19 11:50 AM, akrl@sdf.org wrote:
> Hi all,
> I have a basic question.
> Is there a way to ask libgccjit to parse a conventional .h file?

I have asked a very similar question in 
https://gcc.gnu.org/ml/jit/2015-q2/msg00093.html

> Or alternatively is there a way to have an header file parsed and
> converted in the equivalent libgccjit api calls?
> I ask this because I noticed that, if you jit some code that have to
> inter-operate with non jitted code, maintaining two duplicated definitions
> of all data structures can be quite painful if these are not trivial.
> Alternatively what's the suggested work flow?

I am not sure there is one yet, in practice.

A possible work-around (not entirely trivial) might be to do the 
"opposite": use GCC itself to parse the header file, and write your GCC 
plugin extracting all the relevant information for your particular usage 
of libgccjit. I have no idea how easy that can be for you.

I am not even sure if all the features of C (including some common 
extensions accepted by GCC) are usable from libgccjit. Perhaps bit 
fields and computed gotos like `goto *ptr` and statement expressions 
like `({int x=0; while (y>0) x+=f(y--); x;})` are not easily achievable 
in libgccjit.


Still another thing could be to use LTO: you'll compile your C file with 
GCC using -flto, you'll do LIBGCCJIT things, and the final executable 
sould be compiled and linked with -flto.

Cheers.

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France

Re: about header file parsing

[Marc Nieper-Wißkirchen]

Am Do., 10. Jan. 2019 um 22:18 Uhr schrieb akrl <akrl@sdf.org>:

[...]

> Okay then lets go for gcc_jit_context_add_driver_option then.
>
> >
> >> If we agree on an interface I propose my self to do the patch in
> >> order
> >> to have the occasion to setup the whole process.

Thanks a lot, Andrea.

I am wondering whether you could also make the procedures
playback::context::invoke_embedded_driver and
playback::context::invoke_external_driver available to client code of
libgccjit. (If libgccjit is used as an AOT compiler, the user code may
wish to invoke the driver independently. If the two procedures above
are somehow exposed to user code, it doesn't have to duplicate the
invocation code for an external driver and will be enabled to use the
embedded driver as well.

-- Marc

[...]

Re: about header file parsing

[akrl]

David Malcolm <dmalcolm@redhat.com> writes:

> On Wed, 2019-01-09 at 11:12 +0000, akrl wrote:
>> On  8 January 2019 18:21, David Malcolm <dmalcolm@redhat.com> wrote:
>>
>>     On Tue, 2019-01-08 at 10:50 +0000, akrl@sdf.org wrote:
>>     > Hi all,
>>     > I have a basic question.
>>     > Is there a way to ask libgccjit to parse a conventional .h
>> file?
>>
>>     Sadly, no.
>>
>>     > Or alternatively is there a way to have an header file parsed
>> and
>>     > converted in the equivalent libgccjit api calls?
>>
>>     Having a standard way to do this would be useful.
>>
>>     > I ask this because I noticed that, if you jit some code that
>> have to
>>     > inter-operate with non jitted code, maintaining two duplicated
>>     > definitions
>>     > of all data structures can be quite painful if these are not
>> trivial.
>>
>>     I agree that this is a nuisance.
>>
>>     > Alternatively what's the suggested work flow?
>>
>>     One idea I came up with was to use libabigail:
>>       https://sourceware.org/libabigail/
>>     to generate an ABI description e.g. via abidw:
>>       https://sourceware.org/libabigail/manual/abidw.html
>>     which emits an XML file, and then have a pre-canned way of
>> populating a
>>     gcc_jit_context from such an abigail ABI representation.
>>
>>     This code doesn't exist yet, though (but would just need writing
>> once).
>>
>>     Dave
>>
>>     > Best Regards
>>     >   Andrea
>>     >
>>
>> I think the idea of having something like a gcc plugin that once is
>> parsed
>> compiles the subset of supported C to the necessary libgccjit code
>> would be
>> useful to have for this cases but also educative for learning how to
>> use the
>> library.
>
> I really like this idea.
>
> Code using this plugin is going to want e.g. to access fields of
> structs, globals, etc, so it needs a way to:
> (a) populate a gcc_jit_context with those entities from the parsed C,
> (b) have a way for client code to get at those entities.
>
> As a concrete use-case, I attempted to use libgccjit to write a JIT-
> compiler for CPython, and found myself writing tedious error-prone code
> by hand to express the <Python.h> header:
> https://github.com/davidmalcolm/coconut/blob/master/coconut/compiler.py
>
> (ugh!)
>
> This client code was written in Python, using the Python bindings to
> libgccjit, rather than directly using the <libgccjit.h> C API.
>
> So presumably the plugin ought to support writing out the
> representation in some simple serialized format, and have a way to get
> at it from the various languages that bind libgccjit.
>
> Or maybe it's simplest to start with getting at the reflected IR from
> C?  (I'm thinking aloud here)

I guess would is a good point to start with.

> [Ideally it would support parsing C++ also, but clearly that's some big
> feature-creep: I attempted to write a JIT for GNU Octave, which is
> implemented in C++, and ran into the same issue as above: lots of hand-
> written reflection of the classes and globals.   Probably best to focus
> on a plugin for the C frontend for now.]
>
>>
>> Would be this something that is upstreamable in some form?
>
> Maybe eventually, but given that libgccjit is part of gcc and thus has
> a one-year release cycle, it feels like something that would best be a
> 3rd-party project for now.
>
> Any takers?  (my day job is bug-fixing gcc 9 right now)
>
> Dave
>

Okay I'll try to put some mostly weekend time on it.
I'll come up with something and I'll let you guys know.
Feel free if you have suggestions.

Bests

  Andrea

--
akrl@sdf.org

Re: about header file parsing

[David Malcolm]

On Thu, 2019-01-10 at 19:06 +0000, akrl wrote:

[...]

> Hi,
> today I've wrote a patch that adds GCC_JIT_STR_NEEDED_LIBRARIES to
> gcc_jit_str_option.
> This was to specify a space separated list of the libraries to link
> against.
> I was going to submit the patch for review but this solution seams to
> be
> more general.

FWIW, although it sounds like it's no longer relevant here, I prefer to
avoid extending the
  enum gcc_jit_str_option
and
  enum gcc_jit_int_option
enumerations, in favor of adding new API entrypoints, for the ABI-
detection reason given here:
https://gcc.gnu.org/onlinedocs/jit/topics/contexts.html#options

i.e. "Adding entrypoints for each new option means that client code
that use the new options can be identified directly from binary
metadata, which would not be possible if we instead extended the
various enum gcc_jit_*_option."

> If we agree on an interface I propose my self to do the patch in
> order
> to have the occasion to setup the whole process.
> 
> Bests
> 
>   Andrea

It's great to have new contributors to the project.

Andrea: FWIW, the FSF requires some legal paperwork before I can accept
non-trivial patches; see:
  https://gcc.gnu.org/contribute.html#legal
If you're hoping to send non-trivial patches, it might be an idea to
look into that (unless you already have the FSF paperwork in place?)

Thanks
Dave

Re: about header file parsing

[akrl]

David Malcolm <dmalcolm@redhat.com> writes:

> On Thu, 2019-01-10 at 19:06 +0000, akrl wrote:
>
> [...]
>
>> Hi,
>> today I've wrote a patch that adds GCC_JIT_STR_NEEDED_LIBRARIES to
>> gcc_jit_str_option.
>> This was to specify a space separated list of the libraries to link
>> against.
>> I was going to submit the patch for review but this solution seams to
>> be
>> more general.
>
> FWIW, although it sounds like it's no longer relevant here, I prefer to
> avoid extending the
>   enum gcc_jit_str_option
> and
>   enum gcc_jit_int_option
> enumerations, in favor of adding new API entrypoints, for the ABI-
> detection reason given here:
> https://gcc.gnu.org/onlinedocs/jit/topics/contexts.html#options
>
> i.e. "Adding entrypoints for each new option means that client code
> that use the new options can be identified directly from binary
> metadata, which would not be possible if we instead extended the
> various enum gcc_jit_*_option."

Okay then lets go for gcc_jit_context_add_driver_option then.

>
>> If we agree on an interface I propose my self to do the patch in
>> order
>> to have the occasion to setup the whole process.
>>
>> Bests
>>
>>   Andrea
>
> It's great to have new contributors to the project.
>
> Andrea: FWIW, the FSF requires some legal paperwork before I can accept
> non-trivial patches; see:
>   https://gcc.gnu.org/contribute.html#legal
> If you're hoping to send non-trivial patches, it might be an idea to
> look into that (unless you already have the FSF paperwork in place?)

Yes thanks I'm aware of the process. I'm sorting that out together with
my employer these days.

Bests!
   Andrea

-- 
akrl@sdf.org

about header file parsing

[akrl]

Hi all,
I have a basic question.
Is there a way to ask libgccjit to parse a conventional .h file?
Or alternatively is there a way to have an header file parsed and
converted in the equivalent libgccjit api calls?
I ask this because I noticed that, if you jit some code that have to
inter-operate with non jitted code, maintaining two duplicated definitions
of all data structures can be quite painful if these are not trivial.
Alternatively what's the suggested work flow?

Best Regards
  Andrea

Re: about header file parsing

[akrl]

Basile Starynkevitch <basile@starynkevitch.net> writes:

> I don't know of any GCC plugin within the GCC code base (except the test cases for plugins).
>
> And I am not sure it would be that easy. If I understand well,
> what you suggest would either fill some gcc_jit_context, or create it already filled with an API.
>
> But, AFAIK, there is no current way to query the content of some gcc_jit_context. In other words, it
> seems (but I could be wrong) that what is lacking is an API to e.g. *fetch* all the struct-ures and
> all the functions inside a gcc_jit_context, and we don't have that yet.
>
> Or perhaps, I have misundertood your idea.
>
> My feeling was that the GCC plugin would fill whatever data structure is used by the application using libgccjit.
> And that is application specific -so the GCC plugin has to be application specific too.
>
> Cheers
>
> PS. I could be very wrong, it is not clear in my head!
>
Sorry I try to explain it better.
I was thinking to a gcc plugin (or potentially something else) that
given a C file produce the libgccjit code necessary to fill a context
with what is in the source file.

Say you want then to use it to solve the code duplication we have seen for
structure definitions. You could use a two step compilation strategy:
- first with the plugin you create the libgccjit code you need to define
  in the jitter the structures.
- second you include it in your codebase and you do your usual compilation.

As I sayid it could also potentially compile not just structs and unions
but also the whole C subset supported.

Bests

  Andrea

--
akrl@sdf.org

Re: about header file parsing

[David Malcolm]

On Thu, 2019-01-10 at 13:46 +0100, Marc Nieper-Wißkirchen wrote:
> Am Do., 10. Jan. 2019 um 09:21 Uhr schrieb Marc Nieper-Wißkirchen
> <marc@nieper-wisskirchen.de>:
> > 
> > Am Di., 8. Jan. 2019 um 22:16 Uhr schrieb Basile Starynkevitch
> > <basile@starynkevitch.net>:
> > 
> > [...]
> > 
> > > Still another thing could be to use LTO: you'll compile your C
> > > file with
> > > GCC using -flto, you'll do LIBGCCJIT things, and the final
> > > executable
> > > sould be compiled and linked with -flto.
> > 
> > Is using:
> > 
> > gcc_jit_context_add_command_line_option (ctxt, "-flto helper.o");
> > 
> > supported to have the object generated by libgccjit statically
> > linked
> > with the object helper.o before libgccjit emits the final dynamic
> > library? (Let's assume that helper.o has been compiled with -flto.)
> 
> [...]
> 
> Having taken a look at the source code of libgccjit, it seems that
> the
> command line options set by `gcc_jit_context_add_command_line_option'
> are not passed to the driver used to compile the generated 'fake.s':
> https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/jit/jit-playback.c?view=mar
> kup#l2407

I think so.  They're used by playback::context::compile's
make_fake_args and thus used by the in-process compile, but they're
not used by invoke_driver for the conversion from assembler to shared
library.

I can't remember if that's a deliberate omission, or if an oversight on
my part.

> Dave, would you agree to add a patch that reuses the command line
> options set by `gcc_jit_context_add_command_line_option' when the
> driver is invoked?

It depends on the patch :)

I'm not opposed on principle; gcc_jit_context_add_command_line_option 
already has the caveat:
  "Note that only some options are likely to be meaningful"
in the docs, and it's already something of a loophole for doing things
that I haven't though of.

I wonder if we'd need to have some distinction between options for the
compiler vs options for the driver (maybe introduce a new entrypoint
for driver options?).  IIRC cc1 will complain with a fatal error if it
gets driver-specific options.

> In particular, one would want to be able to
> override `-fno-use-linker-plugin'.
> 
> Here is a use case. Assume that jitted code wants to use <obstack.h>
> (just to give an example). The struct obstack is opaque, and many
> "functions" are implemented as macros. To solve the issues raised in
> this thread, I would write a jit-obstack.c like the following:
> 
> #include <stdalign.h>
> #include <xalloc.h>
> #include <obstack.h>
> 
> #define obstack_chunk_alloc xmalloc
> #define obstack_chunk_free free
> 
> struct jit_obstack
> {
>   alignas (alignof (struct obstack)) char p[sizeof (struct obstack)];
> };
> 
> int
> jit_obstack_init (struct jit_obstack *op) __attribute__ ((visibility
> ("hidden")))
> {
>   return obstack_init ((struct obstack *) op);
> }
> ...
> 
> The jitted code would then access obstacks solely through
> jit-obstack.c. By doing so, we do not have to look at the internals
> of
> <obstack.h> (which may be different on the programmer's and the
> target
> system!) and we do not have to recreate opaque structures in a
> gcc_jit_context.
> 
> However, the code will only run as fast as C directly including
> <obstack.h> if and only if we can use link-time optimization when
> creating the jitted shared object through libgccjit.

It sounds like you want have LTO against a "jit-obstack.so" (or
somesuch) when the driver converts jit-generated assembler to its .so

This sounds really interesting; I've never tried LTO with libgccjit. 
It might just work, but I suspect you might run into unexpected issues
of some kind.  Seems like a fun thing to experiment with.

Dave

Re: about header file parsing

[Basile Starynkevitch]

On 1/9/19 4:52 PM, Marc Nieper-Wißkirchen wrote:
> How would you handle, for example, gmp's header file?
> https://gmplib.org/repo/gmp/file/default/gmp-h.in
>
> Almost all of the public API is behind #define's. So even if the
> gcc_jit_context is filled with the functions that are declared in the
> header, one still needs to know the ABI of GMP to know which functions
> to call.
>
> In order to make it work in the general case (if this is at all
> possible), I think one has to turn the C frontend of GCC into a
> library (like libgccjit has turned the GCC middle/backend into a
> library). This hypothetical library X has its own context K that is
> populated by expanding and parsing the relevant header files.
>
> In this context K, we can then ask for expansion of strings of C
> tokens (e.g. in expression or statement context). Library X then
> returns the expansion in term of, say, a C-specific GENERIC tree,
> which can then be converted into/wrapped into a gcc_jit_object.
>
> -- Marc



AndI (Basile) wrote:

>>> in practice C macros are very important (and a lot of API for C are
>>> using macros, see also macros such as WIFEXITED in wait(2) on some
>>> Linux
>>> system).
>>>
>>> We have to admit that C and libgccjit are different "languages" (even
>>> if
>>> conceptually very close). And a C API is not exactly a set of
>>> "objects"
>>> that libgccjit can handle.
>> They're not the same, but it is important that libgccjit be able to
>> interoperate well with C APIs.


I'm not sure that general goal is reasonable. libgccjit is already able 
to interoperate with a library written in C. Unfortunately, a C API is 
not just a library, but also macros (such as EOF or WIFEXITED), and we 
cannot expect libgccjit to deal with these.


On Linux libgccjit is already capable to call any function in some ELF 
library which uses the C calling conventions, and that is already excellent.

And that limitation is not particular to libgccjit: any implementor of 
some interpreter or compiler has to deal specially with C macros, and 
handle them in an ad-hoc way.

We have to admit that in general C APIs are complex beasts, even if most 
of the time they are "simple" enough.

Regards

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France

Re: about header file parsing

[Marc Nieper-Wißkirchen]

Am So., 13. Jan. 2019 um 20:56 Uhr schrieb akrl <akrl@sdf.org>:

[...]

> Hi Marc,
> there's no technical problem into including these into the patch.
> But unless we have some special reason for that (like minimizing
> LIBGCCJIT_ABI changes) I would rather prefer to have this changes as a
> following patch.
> But I guess is good to have Dave's opinion on all of these points.

Indeed. What is the general policy with respect to the ABI version?
One version bump per patch/feature or one version bump per release
cycle?

[...]

Best,

Marc

Re: about header file parsing

[David Malcolm]

On Wed, 2019-01-09 at 15:04 +0100, Basile Starynkevitch wrote:
> On 1/9/19 12:12 PM, akrl wrote:
> > 
> > I think the idea of having something like a gcc plugin that once is
> > parsed
> > compiles the subset of supported C to the necessary libgccjit code
> > would be
> > useful to have for this cases but also educative for learning how
> > to use the
> > library.
> > 
> > Would be this something that is upstreamable in some form?
> > 
> 
> I don't know of any GCC plugin within the GCC code base (except the
> test cases for plugins).

I think GCC's one-year release would be too slow for such a plugin, so
I think it would best to keep it out-of-tree, at least initially.

> And I am not sure it would be that easy. If I understand well,
> what you suggest would either fill some gcc_jit_context, or create it
> already filled with an API.
> 
> But, AFAIK, there is no current way to query the content of some
> gcc_jit_context. In other words, it
> seems (but I could be wrong) that what is lacking is an API to e.g.
> *fetch* all the struct-ures and
> all the functions inside a gcc_jit_context, and we don't have that
> yet.

Interesting idea, and one that somehow I didn't think of; thanks.

I don't know that client code wants to fetch *all* entities; maybe we
want something like:

extern gcc_jit_type *
gcc_jit_context_get_type_by_name (gcc_jit_context *, const char *);

extern gcc_jit_field *
gcc_jit_type_get_field_by_name (gcc_jit_type *, const char *);

...and so on.  But I'm thinking of this from the "write a JIT for
CPython" use-case, whereas IIRC you're on the functional programming
side of the family.

I think the above would be perfectly implementable, and probably make
it much easier to implement things.

I suppose we could have a:

extern gcc_jit_object *
gcc_jit_context_for_each_object (gcc_jit_context *,
                                 some_callback_type,
                                 void *user_data,
                                 int visit_parent_contexts);

or somesuch, though I don't know if we have enough introspection in the
API yet for this to be useful.

> Or perhaps, I have misundertood your idea.
> 
> My feeling was that the GCC plugin would fill whatever data structure
> is used by the application using libgccjit.
> And that is application specific -so the GCC plugin has to be
> application specific too.

I don't think it has to be; I think the plugin just needs to emit some
kind of reflected representation of the IR of interest in some form
that can populate a gcc_jit_context.

> Cheers
> 
> PS. I could be very wrong, it is not clear in my head!

Thanks

Dave

Re: about header file parsing

[Marc Nieper-Wißkirchen]

Am 08.01.19 um 22:15 schrieb Basile Starynkevitch:
> 
> On 1/8/19 11:50 AM, akrl@sdf.org wrote:
>> Hi all,
>> I have a basic question.
>> Is there a way to ask libgccjit to parse a conventional .h file?
> 
> I have asked a very similar question in 
> https://gcc.gnu.org/ml/jit/2015-q2/msg00093.html
> 
>> Or alternatively is there a way to have an header file parsed and
>> converted in the equivalent libgccjit api calls?
>> I ask this because I noticed that, if you jit some code that have to
>> inter-operate with non jitted code, maintaining two duplicated 
>> definitions
>> of all data structures can be quite painful if these are not trivial.
>> Alternatively what's the suggested work flow?
> 
> I am not sure there is one yet, in practice.
> 
> A possible work-around (not entirely trivial) might be to do the 
> "opposite": use GCC itself to parse the header file, and write your GCC 
> plugin extracting all the relevant information for your particular usage 
> of libgccjit. I have no idea how easy that can be for you.
> 
> I am not even sure if all the features of C (including some common 
> extensions accepted by GCC) are usable from libgccjit. Perhaps bit 
> fields and computed gotos like `goto *ptr` and statement expressions 
> like `({int x=0; while (y>0) x+=f(y--); x;})` are not easily achievable 
> in libgccjit.
> 
> 
> Still another thing could be to use LTO: you'll compile your C file with 
> GCC using -flto, you'll do LIBGCCJIT things, and the final executable 
> sould be compiled and linked with -flto.

What would you do with macros defined in the header file that form part 
of the API? They won't be visible after compiling (already after 
preprocessing).

-- Marc

> 
> Cheers.
> 

Re: about header file parsing

[Marc Nieper-Wißkirchen]

Am Fr., 18. Jan. 2019 um 21:10 Uhr schrieb akrl <akrl@sdf.org>:

[...]

> Okay I've submitted the patch for exposing gcc_jit_context_add_driver_option.

That's great!

Have you experimented with your patch and checked whether linking
object statically with link-time optimization to the
libgccjit-generated code works?

> If Dave's agree on the idea of the modification I'll have a following
> patch for exposing invoke_embedded_driver and invoke_external_driver
> after the review process of the first is done.

Very cool!

[...]

Thanks,

Marc

Re: about header file parsing

[akrl]

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> [...]
>
>> > Dave, would you agree to add a patch that reuses the command line
>> > options set by `gcc_jit_context_add_command_line_option' when the
>> > driver is invoked?
>>
>> It depends on the patch :)
>>
>> I'm not opposed on principle; gcc_jit_context_add_command_line_option
>> already has the caveat:
>>   "Note that only some options are likely to be meaningful"
>> in the docs, and it's already something of a loophole for doing things
>> that I haven't though of.
>>
>> I wonder if we'd need to have some distinction between options for the
>> compiler vs options for the driver (maybe introduce a new entrypoint
>> for driver options?).  IIRC cc1 will complain with a fatal error if it
>> gets driver-specific options.
>
> This is probably the saner approach. Having two entrypoints is
> definitely no less powerful than having just one.
>
> What do you think of gcc_jit_context_add_driver_option? Or,
> alternatively, gcc_jit_context_compile_with_options and
> gcc_jit_context_compile_to_file_with_options?
>
>> > In particular, one would want to be able to
>> > override `-fno-use-linker-plugin'.
>> >
>> > Here is a use case. Assume that jitted code wants to use <obstack.h>
>> > (just to give an example). The struct obstack is opaque, and many
>> > "functions" are implemented as macros. To solve the issues raised in
>> > this thread, I would write a jit-obstack.c like the following:
>> >
>> > #include <stdalign.h>
>> > #include <xalloc.h>
>> > #include <obstack.h>
>> >
>> > #define obstack_chunk_alloc xmalloc
>> > #define obstack_chunk_free free
>> >
>> > struct jit_obstack
>> > {
>> >   alignas (alignof (struct obstack)) char p[sizeof (struct obstack)];
>> > };
>> >
>> > int
>> > jit_obstack_init (struct jit_obstack *op) __attribute__ ((visibility
>> > ("hidden")))
>> > {
>> >   return obstack_init ((struct obstack *) op);
>> > }
>> > ...
>> >
>> > The jitted code would then access obstacks solely through
>> > jit-obstack.c. By doing so, we do not have to look at the internals
>> > of
>> > <obstack.h> (which may be different on the programmer's and the
>> > target
>> > system!) and we do not have to recreate opaque structures in a
>> > gcc_jit_context.
>> >
>> > However, the code will only run as fast as C directly including
>> > <obstack.h> if and only if we can use link-time optimization when
>> > creating the jitted shared object through libgccjit.
>>
>> It sounds like you want have LTO against a "jit-obstack.so" (or
>> somesuch) when the driver converts jit-generated assembler to its .so
>
> Actually, I think I want to link against jit-obstack.o. Thus my
> earlier question, whether supplying "-fuse-linker-plugin -flto
> jit-obstack.o" to the driver is possible.
>
>> This sounds really interesting; I've never tried LTO with libgccjit.
>> It might just work, but I suspect you might run into unexpected issues
>> of some kind.  Seems like a fun thing to experiment with.
>
> We should do as this would allow wrapper libraries like the from the
> example above and will solve many of the problems raised in this
> thread.
>
> [...]
>
> Marc
>

Hi,
today I've wrote a patch that adds GCC_JIT_STR_NEEDED_LIBRARIES to
gcc_jit_str_option.
This was to specify a space separated list of the libraries to link against.
I was going to submit the patch for review but this solution seams to be
more general.
If we agree on an interface I propose my self to do the patch in order
to have the occasion to setup the whole process.

Bests

  Andrea

-- 
akrl@sdf.org

Re: about header file parsing

[Marc Nieper-Wißkirchen]

[...]

> > Dave, would you agree to add a patch that reuses the command line
> > options set by `gcc_jit_context_add_command_line_option' when the
> > driver is invoked?
>
> It depends on the patch :)
>
> I'm not opposed on principle; gcc_jit_context_add_command_line_option
> already has the caveat:
>   "Note that only some options are likely to be meaningful"
> in the docs, and it's already something of a loophole for doing things
> that I haven't though of.
>
> I wonder if we'd need to have some distinction between options for the
> compiler vs options for the driver (maybe introduce a new entrypoint
> for driver options?).  IIRC cc1 will complain with a fatal error if it
> gets driver-specific options.

This is probably the saner approach. Having two entrypoints is
definitely no less powerful than having just one.

What do you think of gcc_jit_context_add_driver_option? Or,
alternatively, gcc_jit_context_compile_with_options and
gcc_jit_context_compile_to_file_with_options?

> > In particular, one would want to be able to
> > override `-fno-use-linker-plugin'.
> >
> > Here is a use case. Assume that jitted code wants to use <obstack.h>
> > (just to give an example). The struct obstack is opaque, and many
> > "functions" are implemented as macros. To solve the issues raised in
> > this thread, I would write a jit-obstack.c like the following:
> >
> > #include <stdalign.h>
> > #include <xalloc.h>
> > #include <obstack.h>
> >
> > #define obstack_chunk_alloc xmalloc
> > #define obstack_chunk_free free
> >
> > struct jit_obstack
> > {
> >   alignas (alignof (struct obstack)) char p[sizeof (struct obstack)];
> > };
> >
> > int
> > jit_obstack_init (struct jit_obstack *op) __attribute__ ((visibility
> > ("hidden")))
> > {
> >   return obstack_init ((struct obstack *) op);
> > }
> > ...
> >
> > The jitted code would then access obstacks solely through
> > jit-obstack.c. By doing so, we do not have to look at the internals
> > of
> > <obstack.h> (which may be different on the programmer's and the
> > target
> > system!) and we do not have to recreate opaque structures in a
> > gcc_jit_context.
> >
> > However, the code will only run as fast as C directly including
> > <obstack.h> if and only if we can use link-time optimization when
> > creating the jitted shared object through libgccjit.
>
> It sounds like you want have LTO against a "jit-obstack.so" (or
> somesuch) when the driver converts jit-generated assembler to its .so

Actually, I think I want to link against jit-obstack.o. Thus my
earlier question, whether supplying "-fuse-linker-plugin -flto
jit-obstack.o" to the driver is possible.

> This sounds really interesting; I've never tried LTO with libgccjit.
> It might just work, but I suspect you might run into unexpected issues
> of some kind.  Seems like a fun thing to experiment with.

We should do as this would allow wrapper libraries like the from the
example above and will solve many of the problems raised in this
thread.

[...]

Marc

Re: about header file parsing

[Marc Nieper-Wißkirchen]

Am Do., 10. Jan. 2019 um 09:21 Uhr schrieb Marc Nieper-Wißkirchen
<marc@nieper-wisskirchen.de>:
>
> Am Di., 8. Jan. 2019 um 22:16 Uhr schrieb Basile Starynkevitch
> <basile@starynkevitch.net>:
>
> [...]
>
> > Still another thing could be to use LTO: you'll compile your C file with
> > GCC using -flto, you'll do LIBGCCJIT things, and the final executable
> > sould be compiled and linked with -flto.
>
> Is using:
>
> gcc_jit_context_add_command_line_option (ctxt, "-flto helper.o");
>
> supported to have the object generated by libgccjit statically linked
> with the object helper.o before libgccjit emits the final dynamic
> library? (Let's assume that helper.o has been compiled with -flto.)

[...]

Having taken a look at the source code of libgccjit, it seems that the
command line options set by `gcc_jit_context_add_command_line_option'
are not passed to the driver used to compile the generated 'fake.s':
https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/jit/jit-playback.c?view=markup#l2407

Dave, would you agree to add a patch that reuses the command line
options set by `gcc_jit_context_add_command_line_option' when the
driver is invoked? In particular, one would want to be able to
override `-fno-use-linker-plugin'.

Here is a use case. Assume that jitted code wants to use <obstack.h>
(just to give an example). The struct obstack is opaque, and many
"functions" are implemented as macros. To solve the issues raised in
this thread, I would write a jit-obstack.c like the following:

#include <stdalign.h>
#include <xalloc.h>
#include <obstack.h>

#define obstack_chunk_alloc xmalloc
#define obstack_chunk_free free

struct jit_obstack
{
  alignas (alignof (struct obstack)) char p[sizeof (struct obstack)];
};

int
jit_obstack_init (struct jit_obstack *op) __attribute__ ((visibility
("hidden")))
{
  return obstack_init ((struct obstack *) op);
}
...

The jitted code would then access obstacks solely through
jit-obstack.c. By doing so, we do not have to look at the internals of
<obstack.h> (which may be different on the programmer's and the target
system!) and we do not have to recreate opaque structures in a
gcc_jit_context.

However, the code will only run as fast as C directly including
<obstack.h> if and only if we can use link-time optimization when
creating the jitted shared object through libgccjit.

-- Marc

Re: about header file parsing

[David Malcolm]

On Wed, 2019-01-09 at 17:15 +0000, akrl wrote:

[...]

> > > 
> > > Would be this something that is upstreamable in some form?
> > 
> > Maybe eventually, but given that libgccjit is part of gcc and thus
> > has
> > a one-year release cycle, it feels like something that would best
> > be a
> > 3rd-party project for now.
> > 
> > Any takers?  (my day job is bug-fixing gcc 9 right now)
> > 
> > Dave
> > 
> 
> Okay I'll try to put some mostly weekend time on it.
> I'll come up with something and I'll let you guys know.
> Feel free if you have suggestions.

Excellent!  Thanks.

Dave

> Bests
> 
>   Andrea
> 
> --
> akrl@sdf.org

Re: about header file parsing

[Marc Nieper-Wißkirchen]

Am Mi., 9. Jan. 2019 um 17:11 Uhr schrieb akrl <akrl@sdf.org>:
>
> Marc Nieper-Wißkirchen <marc.nieper@gmail.com> writes:
>
> > How would you handle, for example, gmp's header file?
> > https://gmplib.org/repo/gmp/file/default/gmp-h.in
> >
> > Almost all of the public API is behind #define's. So even if the
> > gcc_jit_context is filled with the functions that are declared in the
> > header, one still needs to know the ABI of GMP to know which functions
> > to call.
> >
> > In order to make it work in the general case (if this is at all
> > possible), I think one has to turn the C frontend of GCC into a
> > library (like libgccjit has turned the GCC middle/backend into a
> > library). This hypothetical library X has its own context K that is
> > populated by expanding and parsing the relevant header files.
> >
> > In this context K, we can then ask for expansion of strings of C
> > tokens (e.g. in expression or statement context). Library X then
> > returns the expansion in term of, say, a C-specific GENERIC tree,
> > which can then be converted into/wrapped into a gcc_jit_object.
> >
> > -- Marc
>
> So if I undertand correctly: the problem you are raising about defines is
>  about naming in the sense that to use an API under defines you'll
> need to use the the preprocessed expanded names in libgccjit?

Exactly. (Of course, more complicated APIs in form of C header files
are conceivable.)

> If this is the case I still think the tool would be useful.

What one could do (without having to take a look into the header file,
which is good because the internals are not part of the API) would be
to amend the header file by an extra stub. In the case of the EOF
example, one couid put

static int const jit_eof = EOF;

in this stub. When the header file and the stub have been processed by
our hypothetical tool, the context will have been populated with
`jit_eof' (and we rely on the compiler to optimize away the static
variable).

In the case of gmp.h, where, for example, mpq_numref is a macro, the
stub would contain a definition like:

static inline void jit_mpq_numref (mpq_t rop, const mpq_t op)
{
  mpq_numref (rop, op);
}

And so on...

-- Marc

> Andrea
>
> --
> akrl@sdf.org

Re: about header file parsing

[David Malcolm]

On Wed, 2019-01-09 at 11:12 +0000, akrl wrote:
> On  8 January 2019 18:21, David Malcolm <dmalcolm@redhat.com> wrote:
> 
>     On Tue, 2019-01-08 at 10:50 +0000, akrl@sdf.org wrote:
>     > Hi all,
>     > I have a basic question.
>     > Is there a way to ask libgccjit to parse a conventional .h
> file?
>     
>     Sadly, no.
>     
>     > Or alternatively is there a way to have an header file parsed
> and
>     > converted in the equivalent libgccjit api calls?
>     
>     Having a standard way to do this would be useful.
>     
>     > I ask this because I noticed that, if you jit some code that
> have to
>     > inter-operate with non jitted code, maintaining two duplicated
>     > definitions
>     > of all data structures can be quite painful if these are not
> trivial.
>     
>     I agree that this is a nuisance.
>     
>     > Alternatively what's the suggested work flow?
>     
>     One idea I came up with was to use libabigail:
>       https://sourceware.org/libabigail/
>     to generate an ABI description e.g. via abidw:
>       https://sourceware.org/libabigail/manual/abidw.html
>     which emits an XML file, and then have a pre-canned way of
> populating a
>     gcc_jit_context from such an abigail ABI representation.
>     
>     This code doesn't exist yet, though (but would just need writing
> once).
>     
>     Dave
>     
>     > Best Regards
>     >   Andrea
>     >
> 
> I think the idea of having something like a gcc plugin that once is
> parsed
> compiles the subset of supported C to the necessary libgccjit code
> would be
> useful to have for this cases but also educative for learning how to
> use the
> library.

I really like this idea.

Code using this plugin is going to want e.g. to access fields of
structs, globals, etc, so it needs a way to:
(a) populate a gcc_jit_context with those entities from the parsed C, 
(b) have a way for client code to get at those entities.

As a concrete use-case, I attempted to use libgccjit to write a JIT-
compiler for CPython, and found myself writing tedious error-prone code
by hand to express the <Python.h> header:
https://github.com/davidmalcolm/coconut/blob/master/coconut/compiler.py

(ugh!)

This client code was written in Python, using the Python bindings to
libgccjit, rather than directly using the <libgccjit.h> C API.

So presumably the plugin ought to support writing out the
representation in some simple serialized format, and have a way to get
at it from the various languages that bind libgccjit.

Or maybe it's simplest to start with getting at the reflected IR from
C?  (I'm thinking aloud here)

[Ideally it would support parsing C++ also, but clearly that's some big
feature-creep: I attempted to write a JIT for GNU Octave, which is
implemented in C++, and ran into the same issue as above: lots of hand-
written reflection of the classes and globals.   Probably best to focus
on a plugin for the C frontend for now.]

> 
> Would be this something that is upstreamable in some form?

Maybe eventually, but given that libgccjit is part of gcc and thus has
a one-year release cycle, it feels like something that would best be a
3rd-party project for now.

Any takers?  (my day job is bug-fixing gcc 9 right now)

Dave

> Bests
> 
>   Andrea

Re: about header file parsing

[akrl]

Marc Nieper-Wißkirchen <marc@nieper-wisskirchen.de> writes:

> Am Do., 10. Jan. 2019 um 22:18 Uhr schrieb akrl <akrl@sdf.org>:
>
> [...]
>
>> Okay then lets go for gcc_jit_context_add_driver_option then.
>>
>> >
>> >> If we agree on an interface I propose my self to do the patch in
>> >> order
>> >> to have the occasion to setup the whole process.
>
> Thanks a lot, Andrea.
>
> I am wondering whether you could also make the procedures
> playback::context::invoke_embedded_driver and
> playback::context::invoke_external_driver available to client code of
> libgccjit. (If libgccjit is used as an AOT compiler, the user code may
> wish to invoke the driver independently. If the two procedures above
> are somehow exposed to user code, it doesn't have to duplicate the
> invocation code for an external driver and will be enabled to use the
> embedded driver as well.
>
> -- Marc
>
> [...]
>
Hi Marc,
there's no technical problem into including these into the patch.
But unless we have some special reason for that (like minimizing
LIBGCCJIT_ABI changes) I would rather prefer to have this changes as a
following patch.
But I guess is good to have Dave's opinion on all of these points.

Bests
   Andrea

-- 
akrl@sdf.org

Re: about header file parsing

[David Malcolm]

On Tue, 2019-01-08 at 10:50 +0000, akrl@sdf.org wrote:
> Hi all,
> I have a basic question.
> Is there a way to ask libgccjit to parse a conventional .h file?

Sadly, no.

> Or alternatively is there a way to have an header file parsed and
> converted in the equivalent libgccjit api calls?

Having a standard way to do this would be useful.

> I ask this because I noticed that, if you jit some code that have to
> inter-operate with non jitted code, maintaining two duplicated
> definitions
> of all data structures can be quite painful if these are not trivial.

I agree that this is a nuisance.

> Alternatively what's the suggested work flow?

One idea I came up with was to use libabigail:
  https://sourceware.org/libabigail/
to generate an ABI description e.g. via abidw:
  https://sourceware.org/libabigail/manual/abidw.html
which emits an XML file, and then have a pre-canned way of populating a
gcc_jit_context from such an abigail ABI representation.

This code doesn't exist yet, though (but would just need writing once).

Dave

> Best Regards
>   Andrea
> 

Re: about header file parsing

[Marc Nieper-Wißkirchen]

How would you handle, for example, gmp's header file?
https://gmplib.org/repo/gmp/file/default/gmp-h.in

Almost all of the public API is behind #define's. So even if the
gcc_jit_context is filled with the functions that are declared in the
header, one still needs to know the ABI of GMP to know which functions
to call.

In order to make it work in the general case (if this is at all
possible), I think one has to turn the C frontend of GCC into a
library (like libgccjit has turned the GCC middle/backend into a
library). This hypothetical library X has its own context K that is
populated by expanding and parsing the relevant header files.

In this context K, we can then ask for expansion of strings of C
tokens (e.g. in expression or statement context). Library X then
returns the expansion in term of, say, a C-specific GENERIC tree,
which can then be converted into/wrapped into a gcc_jit_object.

-- Marc

Am Mi., 9. Jan. 2019 um 16:43 Uhr schrieb David Malcolm <dmalcolm@redhat.com>:
>
> On Wed, 2019-01-09 at 16:13 +0100, Basile Starynkevitch wrote:
> > On 1/9/19 4:03 PM, akrl wrote:
> > > Sorry I try to explain it better.
> > > I was thinking to a gcc plugin (or potentially something else)
> > > that
> > > given a C file produce the libgccjit code necessary to fill a
> > > context
> > > with what is in the source file.
> >
> > That could not work in general, because C has macros (think of the
> > `EOF`
> > macro constant in `<stdio.h>`) that have no equivalent in libgccjit.
>
> Although possibly not working "in general" I think it could work for
> most important cases.   A macro in C typically can be handled as either
> a client-side function invocation in libgccjit, or as a libgccjit
> function.
>
> For example, CPython has various reference-counting macros:
>
>  #define Py_XDECREF(op) do { if ((op) == NULL) ; else Py_DECREF(op); } while (0)
>
> and in my "coconut" JIT-compiler for CPython I implement this as a
> libgccjit function:
>
> https://github.com/davidmalcolm/coconut/blob/master/coconut/compiler.py
> #L501
>
> (effectively treating it as a static inline function).
>
> > And
> > in practice C macros are very important (and a lot of API for C are
> > using macros, see also macros such as WIFEXITED in wait(2) on some
> > Linux
> > system).
> >
> > We have to admit that C and libgccjit are different "languages" (even
> > if
> > conceptually very close). And a C API is not exactly a set of
> > "objects"
> > that libgccjit can handle.
>
> They're not the same, but it is important that libgccjit be able to
> interoperate well with C APIs.
>
> Dave

Re: about header file parsing

[Basile Starynkevitch]

On 1/9/19 4:03 PM, akrl wrote:
> Sorry I try to explain it better.

> I was thinking to a gcc plugin (or potentially something else) that 
> given a C file produce the libgccjit code necessary to fill a context
> with what is in the source file.

That could not work in general, because C has macros (think of the `EOF` 
macro constant in `<stdio.h>`) that have no equivalent in libgccjit. And 
in practice C macros are very important (and a lot of API for C are 
using macros, see also macros such as WIFEXITED in wait(2) on some Linux 
system).

We have to admit that C and libgccjit are different "languages" (even if 
conceptually very close). And a C API is not exactly a set of "objects" 
that libgccjit can handle.


Cheers

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France

Re: about header file parsing

[David Malcolm]

On Thu, 2019-01-10 at 19:23 +0100, Marc Nieper-Wißkirchen wrote:
> [...]
> 
> > > Dave, would you agree to add a patch that reuses the command line
> > > options set by `gcc_jit_context_add_command_line_option' when the
> > > driver is invoked?
> > 
> > It depends on the patch :)
> > 
> > I'm not opposed on principle;
> > gcc_jit_context_add_command_line_option
> > already has the caveat:
> >   "Note that only some options are likely to be meaningful"
> > in the docs, and it's already something of a loophole for doing
> > things
> > that I haven't though of.
> > 
> > I wonder if we'd need to have some distinction between options for
> > the
> > compiler vs options for the driver (maybe introduce a new
> > entrypoint
> > for driver options?).  IIRC cc1 will complain with a fatal error if
> > it
> > gets driver-specific options.
> 
> This is probably the saner approach. Having two entrypoints is
> definitely no less powerful than having just one.
> 
> What do you think of gcc_jit_context_add_driver_option? Or,
> alternatively, gcc_jit_context_compile_with_options and
> gcc_jit_context_compile_to_file_with_options?

I prefer gcc_jit_context_add_driver_option.

> > > In particular, one would want to be able to
> > > override `-fno-use-linker-plugin'.
> > > 
> > > Here is a use case. Assume that jitted code wants to use
> > > <obstack.h>
> > > (just to give an example). The struct obstack is opaque, and many
> > > "functions" are implemented as macros. To solve the issues raised
> > > in
> > > this thread, I would write a jit-obstack.c like the following:
> > > 
> > > #include <stdalign.h>
> > > #include <xalloc.h>
> > > #include <obstack.h>
> > > 
> > > #define obstack_chunk_alloc xmalloc
> > > #define obstack_chunk_free free
> > > 
> > > struct jit_obstack
> > > {
> > >   alignas (alignof (struct obstack)) char p[sizeof (struct
> > > obstack)];
> > > };
> > > 
> > > int
> > > jit_obstack_init (struct jit_obstack *op) __attribute__
> > > ((visibility
> > > ("hidden")))
> > > {
> > >   return obstack_init ((struct obstack *) op);
> > > }
> > > ...
> > > 
> > > The jitted code would then access obstacks solely through
> > > jit-obstack.c. By doing so, we do not have to look at the
> > > internals
> > > of
> > > <obstack.h> (which may be different on the programmer's and the
> > > target
> > > system!) and we do not have to recreate opaque structures in a
> > > gcc_jit_context.
> > > 
> > > However, the code will only run as fast as C directly including
> > > <obstack.h> if and only if we can use link-time optimization when
> > > creating the jitted shared object through libgccjit.
> > 
> > It sounds like you want have LTO against a "jit-obstack.so" (or
> > somesuch) when the driver converts jit-generated assembler to its
> > .so
> 
> Actually, I think I want to link against jit-obstack.o. Thus my
> earlier question, whether supplying "-fuse-linker-plugin -flto
> jit-obstack.o" to the driver is possible.

Aha.  Makes sense.  I think this would be three invocations of
gcc_jit_context_add_driver_option, one per option.

> > This sounds really interesting; I've never tried LTO with
> > libgccjit.
> > It might just work, but I suspect you might run into unexpected
> > issues
> > of some kind.  Seems like a fun thing to experiment with.
> 
> We should do as this would allow wrapper libraries like the from the
> example above and will solve many of the problems raised in this
> thread.

Indeed (assuming no unexpected snags...)

Dave

Re: about header file parsing

[akrl]

On  8 January 2019 18:21, David Malcolm <dmalcolm@redhat.com> wrote:

    On Tue, 2019-01-08 at 10:50 +0000, akrl@sdf.org wrote:
    > Hi all,
    > I have a basic question.
    > Is there a way to ask libgccjit to parse a conventional .h file?
    
    Sadly, no.
    
    > Or alternatively is there a way to have an header file parsed and
    > converted in the equivalent libgccjit api calls?
    
    Having a standard way to do this would be useful.
    
    > I ask this because I noticed that, if you jit some code that have to
    > inter-operate with non jitted code, maintaining two duplicated
    > definitions
    > of all data structures can be quite painful if these are not trivial.
    
    I agree that this is a nuisance.
    
    > Alternatively what's the suggested work flow?
    
    One idea I came up with was to use libabigail:
      https://sourceware.org/libabigail/
    to generate an ABI description e.g. via abidw:
      https://sourceware.org/libabigail/manual/abidw.html
    which emits an XML file, and then have a pre-canned way of populating a
    gcc_jit_context from such an abigail ABI representation.
    
    This code doesn't exist yet, though (but would just need writing once).
    
    Dave
    
    > Best Regards
    >   Andrea
    >

I think the idea of having something like a gcc plugin that once is parsed
compiles the subset of supported C to the necessary libgccjit code would be
useful to have for this cases but also educative for learning how to use the
library.

Would be this something that is upstreamable in some form?

Bests

  Andrea

Re: about header file parsing

[Marc Nieper-Wißkirchen]

Am 08.01.19 um 22:45 schrieb Basile Starynkevitch:
> Hello all
> 
> Sorry, I have sent a private message to Marc which was meant to be on 
> this jit@gcc.gnu.org list. Here is it for all
> 
> 
> 
> On 1/8/19 10:30 PM, Marc Nieper-Wißkirchen wrote:
>> What would you do with macros defined in the header file that form 
>> part of the API? They won't be visible after compiling (already after 
>> preprocessing). 
> 
> That is one of the problems. Notice that libgccjit don't know about 
> macros, and there is no way to use a macro from libgccjit.
> For example, libgccjit does not know that EOF is -1 on my system.

This seems to be less of a problem if you are able to include the header 
file defining, say, EOF, in your C program that calls the libgccjit 
functions.

> Another problem is inlined functions appearing in headers. They won't be 
> inlined from LIBGCCJIT generated code, unless you use LTO.
> 
> LIBGCCJIT code can only call C-like functions - without inlining them 
> (even when GCC would).
> 
> Perhaps you are only interested in using nice C headers that don't 
> define any inlined functions and have only very few macro constants.
> For the macro constants (like EOF) you would handle them by hand (e.g. 
> expanding the EOF of your Scheme dialect into -1 before using libgccjit).

Ah, yes, this is what I meant above. :)

> 
> BTW, CLASP (see https://github.com/clasp-developers/clasp ...) might be 
> inspirational for you, but doing the equivalent in your case is a lot of 
> work
> because what DrMeister did in CLASP is a lot of work; see also 
> https://drmeister.wordpress.com/ and 
> @Inproceedings{Schafmeister:2015:CLASP,
> author = {Schafmeister, Christian E.},
> booktitle = {Proceedings of the 8\textsuperscript{th} European
> Lisp Symposium on European Lisp Symposium},
> series = {ELS2015},
> year = 2015,
> title = {{CLASP} - A {Common Lisp} that Interoperates with
> {C++} and Uses the {LLVM} Backend}
> }
> 
> 
> I don't understand if you want to handle a specific set of a few header 
> files in your Scheme using libgccjit, or if you want a generic solution.

I am not the original author of the question. I just wanted to remark 
that parsing header files is even more involved.

> If you only want to interace a specific library to your Scheme above 
> LibGccJit, you could use the GCC plugin technique (then hand-tune the few
> problematic areas). If you want a generic solution, it probably won't be 
> enough.
> 
> Cheers.
> 

about header file parsing

[Basile Starynkevitch]

Hello all

Sorry, I have sent a private message to Marc which was meant to be on 
this jit@gcc.gnu.org list. Here is it for all



On 1/8/19 10:30 PM, Marc Nieper-Wißkirchen wrote:
> What would you do with macros defined in the header file that form 
> part of the API? They won't be visible after compiling (already after 
> preprocessing). 

That is one of the problems. Notice that libgccjit don't know about 
macros, and there is no way to use a macro from libgccjit.
For example, libgccjit does not know that EOF is -1 on my system.

Another problem is inlined functions appearing in headers. They won't be 
inlined from LIBGCCJIT generated code, unless you use LTO.

LIBGCCJIT code can only call C-like functions - without inlining them 
(even when GCC would).

Perhaps you are only interested in using nice C headers that don't 
define any inlined functions and have only very few macro constants.
For the macro constants (like EOF) you would handle them by hand (e.g. 
expanding the EOF of your Scheme dialect into -1 before using libgccjit).

BTW, CLASP (see https://github.com/clasp-developers/clasp ...) might be 
inspirational for you, but doing the equivalent in your case is a lot of 
work
because what DrMeister did in CLASP is a lot of work; see also 
https://drmeister.wordpress.com/ and @Inproceedings{Schafmeister:2015:CLASP,
author = {Schafmeister, Christian E.},
booktitle = {Proceedings of the 8\textsuperscript{th} European
Lisp Symposium on European Lisp Symposium},
series = {ELS2015},
year = 2015,
title = {{CLASP} - A {Common Lisp} that Interoperates with
{C++} and Uses the {LLVM} Backend}
}


I don't understand if you want to handle a specific set of a few header 
files in your Scheme using libgccjit, or if you want a generic solution.
If you only want to interace a specific library to your Scheme above 
LibGccJit, you could use the GCC plugin technique (then hand-tune the few
problematic areas). If you want a generic solution, it probably won't be 
enough.

Cheers.

-- 
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France