Does a FFI_STRUCT type really need the elements array?

Does a FFI_STRUCT type really need the elements array?

[Kaz Kylheku]

Hi libffi users,

Here is the thing. Suppose I have a sophisticated FFI implementation
which uses libffi.  Of course I've done all the struct layout 
calculations
yourself: the offset of every member, taking into account its alignment 
to
generate any padding in between them and at the end. (I need this in 
situations
not involving libffi!)

Yet, I have to mirror this information in the libffi data structure:
a FFI_STRUCT type has to have an elements[] array of types. Presumably
libffi just walks these, redundantly doing the same calculations to get
the offset of every member, total size with padding and alignment.

Can't we dispense with the elements[] array and just punch in the size
and alignment values into the struct type (that are supposed to be left
zero?)

Or is there some compromise: a one-element elements[] array (well two, 
with
null termination), where a single element indicates the size and
alignment somehow.

Or how about a one-element array whose one and only pointer is the
null terminator; with the size and alignment punched in: would that
work?

It's wasteful to allocate, initialize and free these arrays and to have
libffi walking over them.

Any words of wisdom?

Re: Does a FFI_STRUCT type really need the elements array?

[Anthony Green]

Hi Kaz,

There are complexities around dealing with structs with nested struct elements.

It sounds like yours is a special case, where you are doing a lot of
work that is normally done by libffi, which is too bad.

The libffi API is far from perfect, but this is one area where there
have been few complaints.  That being said, it would be interesting to
hear a more fully formed proposal that could be considered prior to a
major API breaking release.

AG


On Wed, Jun 21, 2017 at 10:34 PM, Kaz Kylheku <kaz@kylheku.com> wrote:
> Hi libffi users,
>
> Here is the thing. Suppose I have a sophisticated FFI implementation
> which uses libffi.  Of course I've done all the struct layout calculations
> yourself: the offset of every member, taking into account its alignment to
> generate any padding in between them and at the end. (I need this in
> situations
> not involving libffi!)
>
> Yet, I have to mirror this information in the libffi data structure:
> a FFI_STRUCT type has to have an elements[] array of types. Presumably
> libffi just walks these, redundantly doing the same calculations to get
> the offset of every member, total size with padding and alignment.
>
> Can't we dispense with the elements[] array and just punch in the size
> and alignment values into the struct type (that are supposed to be left
> zero?)
>
> Or is there some compromise: a one-element elements[] array (well two, with
> null termination), where a single element indicates the size and
> alignment somehow.
>
> Or how about a one-element array whose one and only pointer is the
> null terminator; with the size and alignment punched in: would that
> work?
>
> It's wasteful to allocate, initialize and free these arrays and to have
> libffi walking over them.
>
> Any words of wisdom?
>

Re: Does a FFI_STRUCT type really need the elements array?

[Kaz Kylheku]

On 22.06.2017 05:01, Anthony Green wrote:
> Hi Kaz,
> 
> There are complexities around dealing with structs with nested struct 
> elements.
> 
> It sounds like yours is a special case, where you are doing a lot of
> work that is normally done by libffi, which is too bad.

Hi Anthony,

There is nothing too bad about it, the work of doing the struct member
offset and alignment calculations are quite a small and easy subtask,
relative to everything. Most of that "everything" is out of the scope
of libffi.  Those little calculations for offsets and whatnot aren't
convenient if they have to be in a parallel framework of objects in
a separate library.

The main value in libffi is that it provides hand-coded machine language
routines for doing call dispatch.

Memory layouts can be done in portable C easily.

Oh, and another requirement I have is that conversion between
C objects and objects in my higher level language has to be supported
even if libffi is not available (isn't detected by the configure 
script).

I have a "HAVE_LIBFFI" preprocessor symbol, such that everything still
builds when it evaluates to zero. The foreign call/callback capability
isn't there, but other features are, like reading a C struct from a 
file,
for instance.

Why I wrote yesterday's posting is that I started adding support for
unions, and realized that the elements[] array thing is getting in the
way. I cannot fill that array with the union element types; libffi
will miscalculate the size. Basically, the approach I'm going to take
is to calculate the size based on the largest union element, then pad
it according to the worst-case alignment, and stick those values into
the ffi_type, which will be of FFI_TYPE_STRUCT. Then I might as well
treat structs and arrays that way too.

> The libffi API is far from perfect, but this is one area where there
> have been few complaints.  That being said, it would be interesting to
> hear a more fully formed proposal that could be considered prior to a
> major API breaking release.

For the people that rely on the struct member calculations, it would
probably be helpful to have these requirements:

* A way to request the calculation to be done after preparing the
   tree of types: like a void ffi_type_init(ffi_type *) function
   or whatever. You shouldn't have to involve a ffi_type in the
   construction of a call descriptor in order to get libffi to
   calculate its size, alignment and member offsets.

* the elements[] array should be an array of structures which
   provide the offset information. (I have exactly such an array in
   my own struct: it holds offsets, and also shift values and
   masks for bitfields.)  For backward compatibility, elements can
   be kept as-is (array of ffi_type * pointers), but a parallel array
   can be added for the extra info. If that array pointer is not null,
   it gets filled in.

* unions could be supported with a FFI_TYPE_UNION which simply
   puts all the elements at offset zero and does the size and
   alignment calculation accordingly.

I get your comment about API breaking. If members are added to
ffi_type, and something like ELF versioning is not used, then
clients compiled with the old headers cannot work with a new lib.

Any types like ffi_type that are allocated by client code should
contain padding members for future extension.

I see in ffi.h that we have this:

typedef struct _ffi_type
{
   size_t size;
   unsigned short alignment;
   unsigned short type;
   struct _ffi_type **elements;
} ffi_type;

which is somewhat of a mistake; there should be something at the end 
like

   void *pad[4]; /* reserved for future extension */

possibly along with the requirement that all correctly written
client code must do memset(&type->pad, 0, sizeof type->pad) or
equivalent, such as using calloc for these objects or whatever.

That requirement makes it possible, in the future, to add
members to the structure whose initialization responsibility
rests with the application. not just members that are managed
internally by the library.

Other possibilities are to have a version handshake;
have a pair of preprocessor symbols FFI_MAJOR and FFI_MINOR which are
passed to some call like ffi_init(FFI_MAJOR, FFI_MINOR).
Then libffi knows exactly which version of the header file
the binary client was compiled with. It knows that an old
client wouldn't have initialized members that do not exist
in its version of the struct declaration. If the space is
reserved, it can be treated as uninitialized, and used.
If the space is not reserved, the library can avoids
touching it.


Cheers ...