Taking the defs file issue

Taking the defs file issue

[Ariel Rios]

Ok guys,

Attached I send the current defs file proposal. I'm ccing also the
gnome-bindings list
for it might be of interest to discuss possible modifications with them
to this
proposal.

ariel

Re: Taking the defs file issue

[Rob Browning]

(I've included gnucash-devel at this point because several of the
 people there should be seeing this discussion, and until I get the
 g-wrap list set up, which should be in about a week and a half,
 gnucash-devel is the de-facto g-wrap devel list.  I'll also CC your
 previous mail there.)

Ariel Rios <ariel@arcavia.com> writes:

> (module module-name
>   (submodule-of module-name)) ;; submodule is optional
> 
> Ex: (module Gtk)
> Ex: (module Rgb
>       (submodule-of Gdk))
> 
> modules are later referred to with a list of module names, like 
> (Gdk Rgb) or (Gtk)
> 
> Object and boxed type definitions automatically create a submodule.
> For example, GtkCList creates the module (module CList (submodule-of
> (Gtk))) which is referred to as module (Gtk CList).

OK, so this is a bit more complex than what I'd gathered from just
looking at the current .defs files.

> (type
>  (alias some-unique-identifier)
>  (in-module module-name)   ;; optional, gchar* is not in a module
>  (gtk-type-id gtk-type-system-id) ;; optional, absent if this is not
>                                   ;; in the type system
>  (is-parametric boolean)          ;; optional default to #f
>  (in-c-name name-of-symbol-in-C)
>  (out-c-name name-of-symbol-in-C)
>  (inout-c-name name-of-symbol-in-C))

What exactly does "parametric" mean in this context?

> Ex: (type
>      (alias string)
>      (gtk-type-id GTK_TYPE_STRING)

Hmm.  gtk-type-id doesn't sound like something that g-wrap would
normally care about, at least not directly since it's very GTK/GNOME
specific.  Is this just used to refer to the macro that should be used
for coercions?

>      (in-c-name "const gchar*")
>      (out-c-name "gchar**")      ;; actually I'm not sure how strings work out/inout
>      (inout-c-name "gchar*"))

How are these used?

After looking at the rest of this document without (admittedly) having
enough time to *carefully* understand everything, though I think I
have a good overall idea of what's going on, it looks like your .defs
file is trying to capture a lot more information than g-wrap does (at
least currently).

Since g-wrap's approach is (I think?) much simpler (and more limited),
perhaps it would be easier for me to summarize g-wrap's approach and
then you (collectively) can tell me whether or not g-wrap's likely to
be of use to you -- either by being enhanced to cover what you need,
or perhaps by being used as a lower-level, more primitive backend that
you generate output for.

Here goes:

From the user's perspective g-wrap knows about three "types of type"
for a wrapped function's arguments and return values (actually four if
you include the enum support I'm about to add): simple-types,
complex-types and pointer-tokens.

Simple types are those for which it's sufficient to just tell g-wrap
how to identify and convert instances of the type.  Usually this means
only types where memory allocation semantics are irrelevant: numeric
types, characters, const-strings, etc.  As an example of a
simple-type, here's how we just added support for "long long" to
g-wrap:

  (add-type 'long-long "long long"  
            ;; fn-convert-to-scm 
            (lambda (x) (list "gh_longlong2scm(" x ")"))
            ;; fn-convert-from-scm 
            (lambda (x) (list "gh_scm2longlong(" x ")"))
            ;; fn-scm-is-a
            (lambda (x) (list "gh_exact_p(" x ")")))

These add-type calls can be added to g-wrap's built in type spec, or
the user can put them in their own files.

complex-types are those that can still be converted to/from a guile
type, but for which memory allocation issues are important.  In those
cases, you have to tell g-wrap, in addition to the above information,
both how to clean up (deallocate) an instance of the type and whether
or not cleaning up those values should be the default for instances of
the type that are passed as arguments, and for instances returned as
return values.  The default can also be overridden on a
per-wrapped-function signature basis.

As an example, here's how you could define a hypothetical type
representing "strings that should be considered to be owned by the
caller", meaning that g-wrap should, by default, "clean up" the C-side
temporary that it creates to use as a parameter to a C-side function
call, or the C-side "temporary" that's received the return value from
the C-side function call.  Also, as you can see from the C-side
conversion code given in the definition below, the new type,
caller-owned-string, also accepts and returns #f as a representation
of the NULL string.

  (add-new-type 
   'caller-owned-string
   (make-complex-c-type
    "const char*"
    ;;fn-convert-to-scm 
    (lambda (x) (list "((" x ") ? gh_str02scm(" x ") : SCM_BOOL_F)"))
    ;;fn-convert-from-scm 
    (lambda (x)
      (list "(((" x ") == SCM_BOOL_F) ? NULL : gh_scm2newstr(" x ", NULL))"))
    ;;fn-scm-is-a
    (lambda (x)
      (list "((" x " == SCM_BOOL_F) || "
            "(SCM_NIMP(" x ") && SCM_STRINGP(" x ")))"))
    ;; c-cleanup-arg-default?
    #t
    ;; c-cleanup-ret-default?
    #t
    ;; fn-c-cleanup
    (lambda (x) (list "if(" x ") { free(" x "); }"))))

That covers simple and complex types where there's a direct
scheme-side representation of the type, but for any types that don't
really have a direct scheme-side representation, g-wrap has
pointer-tokens.  On the scheme side, pointer-tokens appear as an
opaque "holder" for a C-side pointer.  In reality, a pointer-token is
just a smob containing both an indicator of what kind of pointer is
being contained and the C-side pointer itself.  You can't really do
anything with pointer-tokens on the scheme side except pass them
around to other wrapped functions, but in nearly all, if not all the
cases we've had in gnucash, this has been sufficient.

In fact, I'm about to add guile-side access to some of the glib
containers like GLists, GSlists, etc., and I'm just going to use
pointer-tokens.  This means that a thousand element GList* (on the
C-side) would just come over as an opaque (few-byte) pointer-token
rather than being, as you might expect, automatically exploded into a
thousand element scheme list of the contained pointer-tokens.

However, I'm also going to provide conversion functions that'll
"explode" a GList pointer-token into a real scheme list of
sub-pointer-tokens when that's what you need, but by not performing
the "explosion" by default we avoid incurring the overhead of
conversion in cases where all you really want to do is pass the GList*
to another function that expects a GList*.

As an example of pointer-tokens, here's how you might wrap up
GtkWindow*'s in g-wrap:

  (make-pointer-token-type 'GtkWindow* "GtkWindow*")
  
so that then you could define a wrapper for gtk_window_set_title like
this:

  (new-function
   'gtk:gtk-window-set-title
   'void "gtk_window_set_title" '((GtkWindow* window) (const-string title))
   "Set the given window's title.")

As I mentioned, on the guile side, a GtkWindow* would just be
represented as a garbage collected smob that "knew" it was a
GtkWindow*.  It would have a printed representation something like
this: "<pt:GtkWindow:0xAF0AD>".  For instances of GtkWindow*, and
indeed for all pointer-tokens, allocation/deallocation semantics are
left up to the API user, as they would be if you were just programming
gtk from C as well.  This is because in general, I tend to favor this
approach to one that tries to hide the allocation semantics.  Like it
or not, when you're wrapping a C API, I think you generally *do* have
to know (and care) about the allocation semantics, and I tend to feel
that trying to automate them too much (beyond the "'cleanup
'no-cleanup" stuff we already have in g-wrap) is just going to hurt
more than help.

> Whenever a type alias can be used, it is also possible to use the
> keyword "native", which implies that the type in question is too
> C-specific to represent. Then a c-declaration will typically be
> available for use.

This sounds something like g-wrap's pointer-token.

>   (enum DirectionType
>     (in-module Gtk)
>     (c-name GtkDirectionType)
>     (value (nick tab-forward) (c-name GTK_DIR_TAB_FORWARD))

You may want to reconsider using strings for the c-name rather than
symbols since given R5RS, you don't know that case will be preserved
when you go from symbol->string.

>       (c-declaration "c-type-and-name")) ;; c-declaration only required
>                                          ;; if the type alias is "native"
>   (varargs #t) ;; has varargs at the end
> )

How do you handle varargs from guile?  Robert and I tried to figure
out a way to do that last week and came to the conclusion that it
wasn't possible without some non-portable assembly nonsense (or
similar) since C doesn't have anything remotely like "apply" for
va_lists; ISTR Robert actually found something in the C FAQ about
this.

=============

Overall, as I somewhat suggested above, and bearing in mind that I
haven't yet *fully* grokked what you're proposing since I don't have
as much familiarity with gtk, gnome, and the current state of
guile-{gnome,gtk} as you do, it sounds like your current .defs
proposal goes far beyond what g-wrap was intended to do, covering
quite a bit more ground.

Some of that ground seems like it just involves adding more detail to
the wrapping of C-side data structures than g-wrap has heretofore
attempted, but some of it sounds like it's just trying to
automagically handle a lot of very glib/gtk/gnome specific bits,
including bits of the object system -- as it stands now, g-wrap knows
nothing of object hierarchies, struct contents, etc.

So where does this leave us?

With respect to aspects of the .defs proposal that argue for a richer
wrapping of C-side data structures, I'd be in favor of augmenting
g-wrap to cover those bits, though I don't want to lose the ability to
just ignore C-side issues (as you can with pointer-tokens) when you
don't care about fancy handling on the scheme side, and all you want
is performance.

One thing I noticed was that you mentioned was handling of signals
(i.e. c-side callbacks).  If we could think of a good way to handle
that generically, then I'd like to add that to g-wrap.  As yet, g-wrap
doesn't have *any* special facilities for dealing with C-side
callbacks (to/from scheme).  In gnucash, we just use the special
g-wrap type SCM which lets you pass a guile pointer straight through
to C and then require you to set up your own handling of the "thunk
pointer" on the C-side with hand-written functions.

On the issue of "wrapping C structures in more detail" -- right now
g-wrap presumes the C API is "function rich".  By that I mean that it
presumes that you can do everything you need with a structure or other
C datatype via the functional API.  This means that if you want to
wrap an API that requires you to access struct members directly, you
have to write a set of helper functions that are then themselves
g-wrappable.  This has it's drawbacks, but it also has the fairly
substantial advantage of keeping the semantics and implementation of
g-wrap simpler than they would be otherwise.


As a final note, I'm wondering if it might be possible, and make more
sense to think of g-wrap as "assembly language", or as the backend,
for the guile-{gtk,gnome} definitions.  g-wrap could remain targeted
at providing "everything necessary" for wrapping generic C APIs for
scheme (guile), but in perhaps a somewhat primitive fashion.  Then
there would be a translator from the .defs files into g-wrap code that
would handle the GNOME/GTK specifcs like the object system and
hierarchy, etc.

Presuming that's even feasible, then I think that could still provide
us lot of mutual benefit, dividing some of the labor, and sharing the
development of the common bits.

Whew.  Sorry for the length...

Thoughts?
-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: Taking the defs file issue

[Marius Vollmer]

Rob Browning <rlb@cs.utexas.edu> writes:

> This is because in general, I tend to favor this approach to one
> that tries to hide the allocation semantics.  Like it or not, when
> you're wrapping a C API, I think you generally *do* have to know
> (and care) about the allocation semantics, and I tend to feel that
> trying to automate them too much (beyond the "'cleanup 'no-cleanup"
> stuff we already have in g-wrap) is just going to hurt more than
> help.

I quite strongly disagree with this.  In my view, the functions
exported to the Scheme side must not be not be `dangerous' in the
sense that a pilot error can not lead to memory corruption, memory
leaks or similar.

This goal might not be always possible to reach, but one should try
very hard before giving up.  I would even go so far as to require that
the C API be changed so that it can be safely wrapped.  (I did this
with Gtk+, incidentally.)

Guile-gtk is not just the automatic generation of glue code.  This is
actually the easy part.  The hard part is to know what code to
generate and how to make sure that it behaves in a sane way.

[ When I say that generating glue is easy I only meant that in the
  context of guile-gtk.  Designing something more ambitious like
  g-wrap is quite challenging. ]

For example, when you have a pointer-token object in Scheme land, how
do you make sure that the pointer in the token stays valid as long as
it can be used from Scheme?  If you can't guarantee this, you should
not export the pointer to Scheme at all.  It must not be possible to
use a invalid pointer-token in Scheme code, even in buggy Scheme code.
A bug in Scheme should never have uncontrollable consequences like
memory corruption.

Taking you GtkWindow* example, how to you propose to handle the
life-time of GtkWindow structures?  Do you also export gtk_object_ref
and gtk_object_unref to Scheme and expect Scheme programmers to use
them?  I don't think that this would be very welcome.

Anyway, the guts and most of the smarts of guile-gtk are actually in
guile-gtk.c (and in Gtk+ itself), not in build-guile-gtk.  We pretty
much know what glue code we want to generate and it would be nice if
we could use a external tool to do so, but the generated code glue
itself and the `feel' of the resulting Scheme API should pretty much
stay the same.  There is room for improvement, of course, especially
when it comes to composite types and multiple return values, but the
quality of the Scheme API must not decrease.  For example, it is
important that guile-gtk knows about the type hierarchy of the
GtkObject types in Gtk+ so that it can do proper type checking.

> How do you handle varargs from guile?  Robert and I tried to figure
> out a way to do that last week and came to the conclusion that it
> wasn't possible without some non-portable assembly nonsense (or
> similar) since C doesn't have anything remotely like "apply" for
> va_lists; ISTR Robert actually found something in the C FAQ about
> this.

You could try libffi or libffcall (I think).  I once did bindings for
libffi but they have rotten away I'm afraid.

> Overall, as I somewhat suggested above, and bearing in mind that I
> haven't yet *fully* grokked what you're proposing since I don't have
> as much familiarity with gtk, gnome, and the current state of
> guile-{gnome,gtk} as you do, it sounds like your current .defs
> proposal goes far beyond what g-wrap was intended to do, covering
> quite a bit more ground.

[small voice] Lest there be a misunderstanding, I don't know fully
what to think of the defs proposal.  The last time I read it, I found
it to be quite superficial and vague.  I'm not pushing for anything in
this department.  In fact, I'm quite happy sitting on an island with
guile-gtk, knowing that it works quite well as it is, and waiting for
whatever ransom might come... ;)

> On the issue of "wrapping C structures in more detail" -- right now
> g-wrap presumes the C API is "function rich".  By that I mean that it
> presumes that you can do everything you need with a structure or other
> C datatype via the functional API.

Yes, and trivial accessor functions could be generated by g-wrap.
Structure members should be exported as functions to Scheme (there is
pretty much no choice anyway), but the needed accessors functions need
not be written by hand.  An interesting thing would be to merge this
with GOOPS, somehow.

Re: Taking the defs file issue

[Rob Browning]

Marius Vollmer <mvo@zagadka.ping.de> writes:

> I quite strongly disagree with this.  In my view, the functions
> exported to the Scheme side must not be not be `dangerous' in the
> sense that a pilot error can not lead to memory corruption, memory
> leaks or similar.

Well, I suppose this depends on your perspective, and what you want.
If you want to create a tool that lets you do from scheme the same
things you can do from C with a given C API, and if you want to
provide a way for people to *quickly* wrap *existing* C APIs without
having to write a lot of additional glue code, then I think it's going
to be *very* difficult to avoid the "dangerous" things you're talking
about.

I guess this is going to run us right smack into the middle of the
inevitable "flexiblilty vs performance vs safety vs user-burden"
argument, where here user-burden means two different things.  

On the one hand, there's the burden on the "wrapper-spec-writer" to
consider.  If you require that nothing dangerous ever be allowed from
scheme, then the wrapper-spec-writer's job's going to be a lot harder,
and depending on the API, performance may likely suffer.  As you
pointed out, you had to change Gtk+ to make it safe, and in general,
requiring universal safety may often require the wrapper-spec-writer
to add "safe functions" on top of an existing C API before publishing
to scheme via g-wrap.

On the other hand there's the burden on the end-user of the scheme
API.  If you require everything to be safe, their life is probably, in
general, easier, that is unless they just want to translate some code
from C to scheme that already uses the C API successfully, but does it
in a way that while perfect for the C API, doesn't match the "safe"
interface quite right.

Not sure what the right answer is, but it doesn't seem clear-cut to
me.  I could make a case either way...

In the end, if we could design it properly, it still might make sense
to take things in layers.  G-wrap could, at the lowest level, provide
some infrastructure that makes generating the correct glue easier,
even if it's not completely safe, then a safe layer could be put on
top of that that fixes up bits that aren't safe, and then only the
safe functions could be published from the resulting module by
default.

However, I still worry that it may be hard to come up with a safe and
efficient scheme side interface for every C API without a lot of
effort, and in some cases, just having the API might be better than
having the perfect one.

I keep thinking of CPAN and how nice it would be to have some of that
stuff available from guile, and then I think about how much less
manpower we have to implement it, and *that* leads me to think about
g-wrap (or something similar) to help automate the process, and
automatically maintain the result...

Don't get me wrong.  I'd love a solution that was safe, efficient, and
easy on the wrapper-writers.  If we can come up with one, I'd be
thoroughly pleased.

FWIW
-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: Taking the defs file issue

[Tom Phillips]

Rob Browning wrote:
> 
> Marius Vollmer <mvo@zagadka.ping.de> writes:
> 
> > I quite strongly disagree with this.  In my view, the functions
> > exported to the Scheme side must not be not be `dangerous' in the
> > sense that a pilot error can not lead to memory corruption, memory
> > leaks or similar.
> 
> Well, I suppose this depends on your perspective, and what you want.
> If you want to create a tool that lets you do from scheme the same
> things you can do from C with a given C API, and if you want to
> provide a way for people to *quickly* wrap *existing* C APIs without
> having to write a lot of additional glue code, then I think it's going
> to be *very* difficult to avoid the "dangerous" things you're talking
> about.

As someone who's been using guile and guile-gtk, I'd like to throw in my
two cents. When I write in Scheme it's for its characteristics as a
functional language. Any time I have to break down and resort to
procedural constructs, or rely exclusively on side effects, the language
loses some appeal. If I also needed to worry about memory allocation and
deallocation, I think I'd be better off using another language, such as
C, for example. I realize that I'm just one programmer, but I believe
Scheme's functional aspects are what make it a good choice for certain
applications.

Tom Phillips

Re: Taking the defs file issue

[Marius Vollmer]

Rob Browning <rlb@cs.utexas.edu> writes:

> Marius Vollmer <mvo@zagadka.ping.de> writes:
> 
> > I quite strongly disagree with this.  In my view, the functions
> > exported to the Scheme side must not be not be `dangerous' in the
> > sense that a pilot error can not lead to memory corruption, memory
> > leaks or similar.
> 
> Well, I suppose this depends on your perspective, and what you want.
> If you want to create a tool that lets you do from scheme the same
> things you can do from C with a given C API, and if you want to
> provide a way for people to *quickly* wrap *existing* C APIs without
> having to write a lot of additional glue code, then I think it's going
> to be *very* difficult to avoid the "dangerous" things you're talking
> about.

Yes, very true.  I guess my main point is that I don't want guile-gtk
to degrade when switrhcing to g-wrap.  That is, the work of avoiding
"dangerous" things has been done (more or less) and I don't want to
lose this.

Well, I might argue that providing a tool that makes it really easy to
do dangerous things is not a good thing, and that we at least should
put up warning signs all over the place that the quick solution is not
necessarily the best solution and that people should spend a moments
thought on how they can make the bindings safer etc.

> As you pointed out, you had to change Gtk+ to make it safe, and in
> general, requiring universal safety may often require the
> wrapper-spec-writer to add "safe functions" on top of an existing C
> API before publishing to scheme via g-wrap.

Yes, and the problem with Gtk+ was that you could not add these safe
functions on top of the existing API.  That would have been nice.

From a broader perspective, makeing an existing API safe for wrapping
from Scheme should almost always be a significant improvemt of the API
itself, independent from Scheme.  A non-trivial client of the API
should have most of the same problems that a Scheme binding has when
it comes to memory management (type issues are probably different).
For example, ref counting in Gtk+ was inadequate, not just for Scheme,
but for any non-trivial program.  You could just not rely on it
because it interacted badly with the functionality of the objects (for
GtkObjects) or wasn't there at all (for GdkFonts etc if I remember
right).

Cleaning up reference counting made the Gtk+ API better in my view,
independent from the fact that I needed the improvement for the Scheme
bindings.

Anyway, we can not always change existing APIs and I got lucky that
the Gtk+ people have listened to me, but I'd still like bring this
topic up.  I also see it as a kind of education.  There are just too
many crappy APIs out there that just don't need to be that way.  API
design is significantly harder than mere coding and I'm unhappy with
most of the APIs.  Just look at libc.  Horrors!  :-)

Ok, well, I guess I don't need to lecture you, but I couldn't help
myself.

> However, I still worry that it may be hard to come up with a safe and
> efficient scheme side interface for every C API without a lot of
> effort, and in some cases, just having the API might be better than
> having the perfect one.

Yes, but when the API allows a safe binding, g-wrap should be prepared
to generate one.  This is probably mostly an issue of documenting what
exactly a safe binding would be.  How to bridge the Scheme / C gap.  I
feel that for all my rambling, I should really sit down and write such
a thing...

Re: Taking the defs file issue

[Rob Browning]

Marius Vollmer <mvo@zagadka.ping.de> writes:

> Yes, very true.  I guess my main point is that I don't want
> guile-gtk to degrade when switrhcing to g-wrap.  That is, the work
> of avoiding "dangerous" things has been done (more or less) and I
> don't want to lose this.

Totally understandable.

> Well, I might argue that providing a tool that makes it really easy to
> do dangerous things is not a good thing, and that we at least should
> put up warning signs all over the place that the quick solution is not
> necessarily the best solution and that people should spend a moments
> thought on how they can make the bindings safer etc.

No argument here.  In the long run, I'd absolutely prefer the really
nice, really clean, really safe solution, though it might be nice to
have the lower-level dangerous stuff around if you ask for it
explicitly somehow (presuming there were cases where that would be
useful).

> Cleaning up reference counting made the Gtk+ API better in my view,
> independent from the fact that I needed the improvement for the
> Scheme bindings.
> 
> Anyway, we can not always change existing APIs and I got lucky that
> the Gtk+ people have listened to me, but I'd still like bring this
> topic up.  I also see it as a kind of education.  There are just too
> many crappy APIs out there that just don't need to be that way.  API
> design is significantly harder than mere coding and I'm unhappy with
> most of the APIs.  Just look at libc.  Horrors!  :-)
> 
> Ok, well, I guess I don't need to lecture you, but I couldn't help
> myself.

All true.  You're preaching to the choir here :>

> Yes, but when the API allows a safe binding, g-wrap should be prepared
> to generate one.  This is probably mostly an issue of documenting what
> exactly a safe binding would be.  How to bridge the Scheme / C gap.  I
> feel that for all my rambling, I should really sit down and write such
> a thing...

Actually, this could be very helpful.  I'd also love to have better
documentation of related guile low-level issues like what do
GH_DEFER/ALLOW_INTS really do, and when do you need to use them?

Also perhaps a discussion of the changes that had to be made to Gtk+
with a commentary on why they helped would be really informative.

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930