Re: Python API (was Re: Build errors)

[David Malcolm <dmalcolm@redhat.com>]

On Wed, 2013-10-23 at 22:20 -0700, Simon Feltman wrote:
> On Wed, Oct 23, 2013 at 9:07 AM, David Malcolm <dmalcolm@redhat.com> wrote:
> > I'm thinking of maybe rewriting the Python bindings to use cffi rather
> > than Cython - that way they would work with PyPy.  As a potential user,
> > any thoughts on that?
> > (alternatively, we could have two different implementations; not sure if
> > that's sane).
> 
> cffi might also make the bindings a bit easier to write/flush out the
> API with as well? I have a little experience with cffi, the obvious
> thing is it will be slower than static bindings. 

I primarily went with Cython due to my familiarity with it (although I
realized as I went along that I'd forgotten most of it :)

> But this doesn't
> really mean anything until it is tested along with the jit compilation
> times in a real project.

I suspect any arguments about speed of Python bindings are moot compared
to the CPython vs PyPy question, cffi enabling the latter.

That said, I've only ever run the libgccjit.so in "debug" mode (GCC has
lots of internal selftests, which get disabled for speed if you
configure with --enable-checking=release iirc), and there's the huge
internal kludge I'm using to convert from assembler to machine code,
which shows up as taking about half the wallclock time.

>  In this regard, I guess flushing out the API
> with whatever you think is easier or initially more benifitial is the
> way to go (this could also mean sticking with cython bindings since
> they already exists). How much larger do you think the API will grow?

Difficult to say.  I'm conscious that so far I've only tried the API
with a method JIT, and I think supporting a tracing JIT may require some
extra things for supporting patching code after-the-fact.

> >> Some additional thoughts on pygccjit: In terms of API for the enums, I
> >> think it would nice to follow more of a hierarchy. So instead of:
> >>     gccjit.FUNCTION_EXPORTED
> >> It could be:
> >>     gccjit.FunctionKind.EXPORTED
> >
> > (nods).   I'm actually thinking of changing that enum so it applies to
> > visibility in general e.g. for globals.  So that specific one might
> > become GCC_JIT_VISIBILITY_EXPORTED or somesuch in the C API (not sold on
> > that yet).  So would that become gccjit.Visibility.EXPORTED ?
> 
> Seems nice to generalize it for usage with globals as well. It
> probably goes without saying but arguments of this type should also be
> named "visibility" or better yet annotated with
> "visibility:gccjit.Visibility" or documented as such.

(nods)

> > I think it's logical for the user to (optionally) specify the signature
> > when creating the function, within the callback, but we need to use it
> > when wrapping the (void*) from the result object.   So I think we need
> > the Context wrapper object to store a dict, mapping from function names
> > to ctypes signatures, which it would then hand off to the Result wrapper
> > object.   The latter could then use this to create the ctypes wrapper
> > around the (void*) in Result.get_code (or maybe "get_callable"?), thus
> > handing a python callable back to the client code - if that makes sense.
> 
> That makes sense, so as noted in the code comments below?:
> 
> def cb(ctxt):
>     sig = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)
>     # the following will store sig in a table on ctxt
>     fn = ctxt.new_function_from_ctypes("foo", sig)
> result = ctxt.compile()  # result needs to store a ref to ctxt
> # result uses sig from ctxt table to contruct ctypes callable
> code = result.get_code(b"square")
> 
> With this scheme, I guess the standard API of ctxt.new_function would
> internally create a CFUNCTYPE with a signature mapped from the gccjit
> types for use as a callable later on.
So it uses the ctypes types to get at gccjit types?  Would need a way to
get at the params also.

> I think "get_callable" or even "get_pycallable" might be a bit more
> understandable to a reader. The "py" prefix, albeit somewhat ugly,
> gives a sense of orientation in regards to a jit construct vs a python
> construct.

Sounds sane.

> One thing I've noticed (this also stood out to me when playing with
> llvmpy in the past) is types need to be created (and bound?) to a
> context. This is well beyond my knowledge of compiler architecture,
> but it seems like this could be cleaner?
> 
>     the_type = ctxt.get_type(gccjit.Type.INT)
>     local_i = fn.new_local(the_type, b"i")
> 
> could be:
>     local_i = fn.new_local(gccjit.Type.INT, b"i")
In the second syntax, is gccjit.Type.INT actually giving back a
gccjit.Type object, calling gcc_jit_get_type under the covers?


> I'm sure there are performance and reference cycle implications with
> this, and would probably convolute the internals. fn would need a
> reference to the context which would need to hold a table of type
> instances, etc.. (actually similar to the function building problem
> above). It also might be too divergent from the C API which can be a
> hard thing to balance with bindings. So I guess it depends how much
> you want to diverge from the C API for Python convience.

(nods)   I'm not sure.

Thanks
Dave