test results on many platforms

test results on many platforms

[Bruno Haible]

Hi,

I tested libffi-3.2.1 (building from source, and then test it using the
test suite from https://haible.de/bruno/gnu/libffi-testsuite.tar.gz ).

I've registered individual bugs in https://github.com/libffi/libffi/issues .

Here's a summary of the results:

aix-powerpc-32-gcc       FTBFS
aix-powerpc-32-xlc       FTBFS
aix-powerpc-64-gcc       FTBFS
aix-powerpc-64-xlc       FTBFS
cygwin-32                failed-call, failed-callback
cygwin-64                failed-call, failed-callback
freebsd-arm64            OK
freebsd-x86-32           test-call crash, failed-callback
freebsd-x86-64           OK
hpux-hppa-32             FTBFS
hpux-hppa-64             FTBFS
hpux-ia64-32             FTBFS
hpux-ia64-64             FTBFS
hurd-x86                 OK
irix-mips-32             FTBFS
irix-mips-n32-cc         FTBFS
irix-mips-n32-gcc        FTBFS
kfreebsd-x86-64          OK
linux-alpha              test-call unaligned access
linux-alpine-x86-64      test-callback crash
linux-arm64              OK
linux-armel              failed-callback
linux-armelhf            FTBFS; failed-callback
linux-centos6-x86-64     OK
linux-centos7-x86-64     OK
linux-hppa               test-call crash
linux-ia64               failed-call, failed-callback
linux-m68k               failed-call, failed-callback
linux-mipseb-32          failed-call, failed-callback
linux-mipseb-64          failed-call, failed-callback
linux-mipseb-n32         failed-call, failed-callback
linux-mipsel-32          failed-callback
linux-mipsel-64          failed-callback
linux-mipsel-n32         failed-callback
linux-powerpc-32         failed-call, failed-callback
linux-powerpc-64         failed-call, failed-callback
linux-powerpc-64el       failed-callback
linux-s390-32            failed-call, failed-callback
linux-s390-64            failed-call, failed-callback
linux-sparc-32           test-call crash, failed-callback
linux-sparc-64           test-call crash, failed-callback
linux-x86-32             OK
linux-x86-64             OK
linux-x86-x32            FTBFS
macosx-powerpc           failed-call, failed-callback
macosx-x86-32            FTBFS
macosx-x86-64            test-callback crash
macosx-x86-64-clang      OK
netbsd-sparc-32          test-call crash, test-callback crash
netbsd-x86-32            OK
netbsd-x86-64            test-callback crash
openbsd-x86-32           test-call crash, failed-callback
openbsd-x86-64           test-call crash
solaris10-sparc-32-cc    failed-call, failed-callback
solaris10-sparc-32-gcc   test-call crash, failed-callback
solaris10-sparc-64-cc    FTBFS
solaris10-sparc-64-gcc   test-call crash, failed-callback
solaris10-x86-32-cc      FTBFS
solaris10-x86-32-gcc     OK
solaris10-x86-64-cc      FTBFS
solaris10-x86-64-gcc     test-callback crash
windows-mingw-32         failed-call, test-callback crash
windows-mingw-64         test-call crash, failed-callback
windows-msvc-32          FTBFS
windows-msvc-64          FTBFS

Legend:
- FTBFS means "fails to build from source".
- "test-call crash" means that the test program that exercises
  ffi_call() crashes.
- failed-call means that ffi_call behaves badly at runtime,
  but no crash.
- "test-callback crash" means that the test program that exercises
  ffi_closure crashes.
- failed-callback means that ffi_closure behaves badly at runtime,
  but no crash.

Best regards,

               Bruno

Re: test results on many platforms

[Matthias Klose]

On 21.10.2017 20:31, Bruno Haible wrote:
> Hi,
> 
> I tested libffi-3.2.1 (building from source, and then test it using the
> test suite from https://haible.de/bruno/gnu/libffi-testsuite.tar.gz ).
> 
> I've registered individual bugs in https://github.com/libffi/libffi/issues .
> 
> Here's a summary of the results:

[...]

could you repeat these tests with libffi trunk?

could these new tests added to the libffi testsuite itself?

Matthias

Re: test results on many platforms

[Bruno Haible]

Hi Matthias,

> could you repeat these tests with libffi trunk?

No, I won't do that.

1) When I test something, I want to test the build infrastructure with it.
   Therefore I take a tarball with 'configure' file as input.

2) You have to understand that it takes time to run tests on dozens of
   machines, for 65 ABIs, and review the results.

I am not the only person who can run these tests:
  - The test suite is at a public place.[1]
  - Many of the platforms are available to anyone, through QEMU emulation.[2]

But if you provide a (release or prerelease) tarball and I see signs that
some of the bugs might be fixed, I might find the time to verify that these
bug are indeed fixed.

> could these new tests added to the libffi testsuite itself?

If you can accommodate a GPLed testsuite in the MIT/BSD-licensed libffi,
you can do that already now.

Otherwise, I can ask my collaborator if it would be ok with him to relicense
this testsuite under the MIT/BSD license used by libffi.

Bruno

[1] https://haible.de/bruno/gnu/libffi-testsuite.tar.gz
[2] http://git.savannah.gnu.org/gitweb/?p=libffcall.git;a=tree;f=porting-tools/emulation

Re: test results on many platforms

[Kaz Kylheku (libffi)]

Hi Bruno,

I suspect that some of your ffi_call bug reports on 64 bits may be 
invalid.

The return value on 64 bits, of small types, requires special treatment
due to known quirk/design flaw in the API. It was originally not 
documented,
and then it was just documented as is. The way your code is doing it
is how it *should* be, but isn't.

Firstly, you must allocate a full "ffi_arg" word for the return value.
You can't just use a &int_variable pointer for the return value of
type int. The "ffi_arg" is in fact a 64 bit word.

Secondly, the return value comes out aligned funny inside this word on
big endian platforms. It is not at the base address as it would be
on little endian, but rather the value of the wider ffi_arg word is
the return value: in other words, the "int" is in the high order bytes
which are four bytes above the base address.

This situation bears some resemblance to promotion in the C language!
Or, rather, not the type promotion from char/short to int as much as
the widening in old-style calls: how char and short arguments are 
actually
int arguments:

    int old_style(arg)
    char arg;
    {
      /*...*/
    }

the argument is really of type int. When we call old_style((char) 42),
promotion dove-tails with this: the char expression promotes to int,
and so the correct call is generated, nonexistence of prototype
notwithstanding.

In libffi return value passage, every integer type smaller than ffi_arg
is "promoted" to ffi_arg.

Naively written code will appear to work fine on little endian 64 bit,
such as x86_64, and then flop on, say, PPC64.

I ran into this in the TXR Lisp FFI, resulting in this commit:

http://www.kylheku.com/cgit/txr/commit/ffi.c?id=10507446389cff9b072c25aa86ba35834a786fe5

My FFI type systems's virtual functions for doing "put" and "get" memory
operations had to be split into "put", "rput", "get" and "rget" with the
"r" variants doing  the right thing for return values. The pad_retval
macro calculates a padded size for any given type if it is used as a
return value.

Many thanks to the helpful hints from this mailing list.

Re: small return types

[Bruno Haible]

Hi Kaz,

Changing the subject, since we're talking about
https://github.com/libffi/libffi/issues/361
https://github.com/libffi/libffi/issues/362
https://github.com/libffi/libffi/issues/368
https://github.com/libffi/libffi/issues/369

> I suspect that some of your ffi_call bug reports on 64 bits may be 
> invalid.
> 
> The return value on 64 bits, of small types, requires special treatment
> due to known quirk/design flaw in the API. It was originally not 
> documented,
> and then it was just documented as is. The way your code is doing it
> is how it *should* be, but isn't.

The 3.2.1 documentation says:

 -- Function: ffi_status ffi_prep_cif (ffi_cif *CIF, ffi_abi ABI,
          unsigned int NARGS, ffi_type *RTYPE, ffi_type **ARGTYPES)
     This initializes CIF according to the given parameters.
     ...
     RTYPE is a pointer to an 'ffi_type' structure that describes the
     return type of the function.  *Note Types::.

Types:
'Libffi' provides a number of built-in type descriptors that can be used
to describe argument and return types:
    ...

The new documentation says:

  +That is, in most cases, @var{ret} points to an object of exactly the
  +size of the type specified when @var{cif} was constructed.  However,
  +integral types narrower than the system register size are widened.

Which is not useful, because the point of using a library such as libffi
is to NOT NEED TO KNOW about the ABI, about the width of system registers
etc.

> This situation bears some resemblance to promotion in the C language!

But this resemblance is not a justification for libffi's behaviour,
because
  1) 'char'. 'unsigned char' etc. are considered as valid return types
     of functions (and different from 'int') since ANSI C, 1989/1990.
  2) C does not do promotion from 32-bit integer types to 64-bit integer
     types.

> Naively written code will appear to work fine on little endian 64 bit,

No, the bug https://github.com/libffi/libffi/issues/368 also affect
some little-endian platforms.

Bruno

Re: small return types

[Kaz Kylheku (libffi)]

On 22.10.2017 13:33, Bruno Haible wrote:
> The new documentation says:
> 
>   +That is, in most cases, @var{ret} points to an object of exactly the
>   +size of the type specified when @var{cif} was constructed.  However,
>   +integral types narrower than the system register size are widened.
> 
> Which is not useful, because the point of using a library such as 
> libffi
> is to NOT NEED TO KNOW about the ABI, about the width of system 
> registers
> etc.

Unfortunately, that's the library interface that a test suite has
to exercise, and not the ideal one that we would like to exist in
it place.

This cannot be fixed without breaking correctly implemented programs
which follow the scheme.

Using the API right is slightly ugly but not a major complication.

My FFI implementation experience shows that it can be absorbed in a
design which didn't anticipate the issue at all, and in a reasonably 
localized
way which hides it from the higher levels.

The register size is abstracted by the ffi_arg type; applications don't
have to depend on how specific sizes in specific ways.

Re: small return types

[Tom Tromey]

>>>>> "Bruno" == Bruno Haible <bruno@clisp.org> writes:

Bruno> Changing the subject, since we're talking about
Bruno> https://github.com/libffi/libffi/issues/361
Bruno> https://github.com/libffi/libffi/issues/362
Bruno> https://github.com/libffi/libffi/issues/368
Bruno> https://github.com/libffi/libffi/issues/369

>> Naively written code will appear to work fine on little endian 64 bit,

Bruno> No, the bug https://github.com/libffi/libffi/issues/368 also affect
Bruno> some little-endian platforms.

It sounds this one is a different bug.  I haven't looked at your test
suite yet, so no speculation from me about what it might be.

For the others, I think we should close them; or perhaps close all but
one of them and change the last one into a libffi 4 wish-list bug.

Kaz's analysis here is, I believe, correct -- libffi is it the way it
is, and the tests should be changed.  Changing libffi is probably a good
idea, but it will also break the current users, so it would be best to
roll this sort of change up with other desirable breaking changes.  And,
as you can probably see, there isn't a huge amount of work going on in
libffi, so it's unclear to me at least whether this could be
accomplished for the existing targets.

Tom