Re: [PATCH v2 0/4] libffi: Sync with upstream

["H.J. Lu" <hjl.tools@gmail.com>]

On Sat, Oct 16, 2021 at 1:07 PM David Edelsohn <dje.gcc@gmail.com> wrote:
>
> On Sat, Oct 16, 2021 at 3:59 PM H.J. Lu <hjl.tools@gmail.com> wrote:
> >
> > On Sat, Oct 16, 2021 at 12:53 PM David Edelsohn <dje.gcc@gmail.com> wrote:
> > >
> > > On Sat, Oct 16, 2021 at 1:13 PM H.J. Lu <hjl.tools@gmail.com> wrote:
> > > >
> > > > On Sat, Oct 16, 2021 at 10:04 AM David Edelsohn <dje.gcc@gmail.com> wrote:
> > > > >
> > > > > On Sat, Oct 16, 2021 at 7:48 AM H.J. Lu <hjl.tools@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Oct 15, 2021 at 5:22 PM David Edelsohn <dje.gcc@gmail.com> wrote:
> > > > > > >
> > > > > > > On Fri, Oct 15, 2021 at 8:06 PM H.J. Lu <hjl.tools@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Oct 13, 2021 at 6:42 AM H.J. Lu <hjl.tools@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Oct 13, 2021 at 6:03 AM Richard Biener
> > > > > > > > > <richard.guenther@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Wed, Oct 13, 2021 at 2:56 PM H.J. Lu <hjl.tools@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Oct 13, 2021 at 5:45 AM Richard Biener
> > > > > > > > > > > <richard.guenther@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 2, 2021 at 5:50 PM H.J. Lu <hjl.tools@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Change in the v2 patch:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Disable static trampolines by default.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > GCC maintained a copy of libffi snapshot from 2009 and cherry-picked fixes
> > > > > > > > > > > > > from upstream over the last 10+ years.  In the meantime, libffi upstream
> > > > > > > > > > > > > has been changed significantly with new features, bug fixes and new target
> > > > > > > > > > > > > support.  Here is a set of patches to sync with libffi 3.4.2 release and
> > > > > > > > > > > > > make it easier to sync with libffi upstream:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Document how to sync with upstream.
> > > > > > > > > > > > > 2. Add scripts to help sync with upstream.
> > > > > > > > > > > > > 3. Sync with libffi 3.4.2. This patch is quite big.  It is availale at
> > > > > > > > > > > > >
> > > > > > > > > > > > > https://gitlab.com/x86-gcc/gcc/-/commit/15e80c879c571f79a0e57702848a9df5fba5be2f
> > > > > > > > > > > > > 4. Integrate libffi build and testsuite with GCC.
> > > > > > > > > > > >
> > > > > > > > > > > > How did you test this?  It looks like libgo is the only consumer of
> > > > > > > > > > > > libffi these days.
> > > > > > > > > > > > In particular go/libgo seems to be supported on almost all targets besides
> > > > > > > > > > > > darwin/windows - did you test cross and canadian configurations?
> > > > > > > > > > >
> > > > > > > > > > > I only tested it on Linux/i686 and Linux/x86-64.   My understanding is that
> > > > > > > > > > > the upstream libffi works on Darwin and Windows.
> > > > > > > > > > >
> > > > > > > > > > > > I applaud the attempt to sync to upsteam but I fear you won't get any "review"
> > > > > > > > > > > > of this massive diff.
> > > > > > > > > > >
> > > > > > > > > > > I believe that it should just work.  Our libffi is very much out of date.
> > > > > > > > > >
> > > > > > > > > > Yes, you can hope.  And yes, our libffi is out of date.
> > > > > > > > > >
> > > > > > > > > > Can you please do the extra step to test one weird architecture, namely
> > > > > > > > > > powerpc64-aix which is available on the compile-farm?
> > > > > > > > >
> > > > > > > > > I will give it a try and report back.
> > > > > > > > >
> > > > > > > > > > If that goes well I think it's good to "hope" at this point (and plenty of
> > > > > > > > > > time to fix fallout until the GCC 12 release).
> > > > > > > > > >
> > > > > > > > > > Thus OK after the extra testing dance and waiting until early next
> > > > > > > > > > week so others can throw in a veto.
> > > > > > > >
> > > > > > > > I tried to bootstrap GCC master branch on  gcc119.fsffrance.org:
> > > > > > > >
> > > > > > > > *  MT/MODEL: 8284-22A                                                         *
> > > > > > > > * Partition: gcc119                                                           *
> > > > > > > > *    System: power8-aix.osuosl.org                                            *
> > > > > > > > *       O/S: AIX V7.2 7200-04-03-2038
> > > > > > > >
> > > > > > > > I configured GCC with
> > > > > > > >
> > > > > > > > --with-as=/usr/bin/as --with-ld=/usr/bin/ld
> > > > > > > > --enable-version-specific-runtime-libs --disable-nls
> > > > > > > > --enable-decimal-float=dpd --disable-libstdcxx-pch --disable-werror
> > > > > > > > --enable-__cxa_atexit --with-gmp=/opt/cfarm --with-mpfr=/opt/cfarm
> > > > > > > > --with-mpc=/opt/cfarm --with-isl=/opt/cfarm --prefix=/opt/freeware
> > > > > > > > --with-local-prefix=/opt/freeware --enable-languages=c,c++,go
> > > > > > > >
> > > > > > > > I got
> > > > > > > >
> > > > > > > > g++   -g -DIN_GCC     -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W
> > > > > > > > -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-at
> > > > > > > > tribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-
> > > > > > > > overlength-strings -fno-common  -DHAVE_CONFIG_H  -DGENERATOR_FILE -static-libstd
> > > > > > > > c++ -static-libgcc -Wl,-bbigtoc -Wl,-bmaxdata:0x40000000 -o build/genenums \
> > > > > > > >     build/genenums.o build/read-md.o build/errors.o ../build-powerpc-ibm-aix7.2.
> > > > > > > > 4.0/libiberty/libiberty.a
> > > > > > > > ld: 0711-317 ERROR: Undefined symbol: lexer_line
> > > > > > > > ld: 0711-317 ERROR: Undefined symbol: .yylex(char const**)
> > > > > > > > ld: 0711-317 ERROR: Undefined symbol: .yybegin(char const*)
> > > > > > > > ld: 0711-317 ERROR: Undefined symbol: lexer_toplevel_done
> > > > > > > > ld: 0711-317 ERROR: Undefined symbol: .yyend()
> > > > > > > > ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
> > > > > > > > collect2: error: ld returned 8 exit status
> > > > > > > > Makefile:3000: recipe for target 'build/gengtype' failed
> > > > > > > > gmake[5]: *** [build/gengtype] Error 1
> > > > > > > >
> > > > > > > > David, is there an instruction to bootstrap GCC on AIX?
> > > > > > >
> > > > > > > The CompileFarm page in the GCC wiki has instructions under "build tips":
> > > > > > >
> > > > > > > https://gcc.gnu.org/wiki/CompileFarm#Services_and_software_installed_on_farm_machines
> > > > > > >
> > > > > > > The error that you show might be due to not having /opt/freeware/bin
> > > > > > > first in your path and the bootstrap used the AIX version of lex or
> > > > > > > sed or some other command.
> > > > > > >
> > > > > >
> > > > > > Hi David,
> > > > > >
> > > > > > I made some progress.  I am trying to verify my libffi sync branch:
> > > > > >
> > > > > > https://gitlab.com/x86-gcc/gcc/-/tree/users/hjl/libffi/master
> > > > > >
> > > > > > on powerpc64-aix.  Go is the only user of libffi.  But compiler farm
> > > > > > suggestion is
> > > > > >
> > > > > > .../src/configure --disable-werror --enable-languages=c,c++
> > > > > > --with-gmp=/opt/cfarm --with-libiconv-prefix=/opt/cfarm
> > > > > > --disable-libstdcxx-pch --with-included-gettext
> > > > > >
> > > > > > When I added Go with --enable-languages=c,c++,go, I got
> > > > > >
> > > > > > /opt/freeware/bin/bash: missing-objcopy: command not found
> > > > > > /opt/freeware/bin/bash: missing-objcopy: command not found
> > > > > > /opt/freeware/bin/bash /home/hjl/work/git/gitlab/x86-gcc/libgo/mvifdiff.sh build
> > > > > > cfg.go.tmp buildcfg.go
> > > > > > mv: 0653-401 Cannot rename internal/unsafeheader.s-gox.tmp to internal/unsafehea
> > > > > > der.gox:
> > > > > >              A file or directory in the path name does not exist.
> > > > > > mv: 0653-401 Cannot rename runtime/internal/atomic.s-gox.tmp to runtime/internal
> > > > > > /atomic.gox:
> > > > > >              A file or directory in the path name does not exist.
> > > > > > mv: 0653-401 Cannot rename internal/race.s-gox.tmp to internal/race.gox:
> > > > > >              A file or directory in the path name does not exist.
> > > > > > Makefile:3019: recipe for target 'runtime/internal/atomic.s-gox' failed
> > > > > > gmake[10]: *** [runtime/internal/atomic.s-gox] Error 1
> > > > > >
> > > > > > Is Go supported on AIX? If yes, what did I do wrong?
> > > > >
> > > > > I believe that GCC Go works on AIX, but I have not been testing it
> > > > > regularly.  There was more focus on GCC Go for AIX prior to the Golang
> > > > > port to AIX.
> > > > >
> > > > > It looks like the problem is that the objcopy command is not
> > > > > installed.  There have been various issues with Binutils on AIX.  I'm
> > > > > not certain if GCC Go had relied on objcopy or not.
> > > > >
> > > >
> > > > Hi Richard,
> > > >
> > > > Go on master branch isn't buildable on AIX.  What should I do
> > > > next to get my libffi sync patches into master branch?
> > >
> > > H.J.,
> > >
> > > You have mischaracterized and misrepresented the issue.  No one stated
> > > that GCC Go is not buildable, only that GCC Go requires objcopy, which
> > > is not currently installed on gcc119.
> > >
> > > Thanks, David
> >
> > I installed GNU binutils and got
> >
> >
> > /home/hjl/work/git/gitlab/x86-gcc/libgo/go/hash/maphash/maphash.go:17:30:
> > error: ./internal/unsafeheader.gox exists but does not contain any Go
> > export data
> >    17 |         "internal/unsafeheader"
> >       |                              ^
> > /home/hjl/work/git/gitlab/x86-gcc/libgo/go/hash/maphash/maphash.go:17:30:
> > error: internal/unsafeheader.gox exists but does not contain any Go
> > export data
> > /home/hjl/work/git/gitlab/x86-gcc/libgo/go/hash/maphash/maphash.go:17:30:
> > error: import file 'internal/unsafeheader' not found
> > /home/hjl/work/git/gitlab/x86-gcc/libgo/go/hash/maphash/maphash.go:146:42:
> > error: reference to undefined name 'unsafeheader'
> >   146 |                         ptr :=
> > (*byte)((*unsafeheader.String)(unsafe.Pointer(&s)).Data)
> >       |                                          ^
> > /home/hjl/work/git/gitlab/x86-gcc/libgo/go/hash/maphash/maphash.go:146:41:
> > error: expected pointer
> >   146 |                         ptr :=
> > (*byte)((*unsafeheader.String)(unsafe.Pointer(&s)).Data)
> >       |                                         ^
> > gmake[8]: Entering directory
> > '/home/hjl/work/build/gnu/tools-build/gcc-gitlab/build-powerpc64-aix7.2/powerpc-ibm-aix7.2.4.0/pthread/libgo'
> > f="internal/itoa.o"; if test ! -f $f; then f="internal/.libs/itoa.o";
> > fi; objcopy -j .go_export $f internal/itoa.s-gox.tmp;
> > /opt/freeware/bin/bash
> > /home/hjl/work/git/gitlab/x86-gcc/libgo/mvifdiff.sh
> > internal/itoa.s-gox.tmp `echo internal/itoa.s-gox | sed -e
> > 's/s-gox/gox/'`
> > echo timestamp > internal/itoa.s-gox
> > Makefile:3019: recipe for target 'hash/maphash.lo' failed
> > gmake[6]: *** [hash/maphash.lo] Error 1
> >
> > and there were more errors like this.
>
> I have asked my colleagues who work on Go for AIX about the status. I
> will let you know when I receive a reply.
>
> libffi trunk works on AIX.
>
> Is Go the only usage of libffi within GCC?  If so, then it seems that

Yes.

> the issue for the AIX build of GCC is to ensure that the update of
> libffi doesn't break the rest of the build and that the in-tree build
> of libffi succeeds.  Does the in-tree build of libffi pass its
> self-tests, or at least the same results as libffi out of the tree?

The test results of the in-tree libffi from libffi 3.4.2 are

FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=12
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=12
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=13
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=13
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=55
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=55
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=56
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=56
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=57
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=57
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=68
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-call.c -W -Wall -Wno-psabi -DDGTEST=68
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test
FAIL: libffi.bhaible/test-callback.c -W -Wall -Wno-psabi -DDGTEST=54
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-callback.c -W -Wall -Wno-psabi -DDGTEST=54
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test
FAIL: libffi.bhaible/test-callback.c -W -Wall -Wno-psabi -DDGTEST=55
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-callback.c -W -Wall -Wno-psabi -DDGTEST=55
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test
FAIL: libffi.bhaible/test-callback.c -W -Wall -Wno-psabi -DDGTEST=56
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O0 execution test
FAIL: libffi.bhaible/test-callback.c -W -Wall -Wno-psabi -DDGTEST=56
-Wno-unused-variable -Wno-unused-parameter
-Wno-unused-but-set-variable -Wno-uninitialized -O2 execution test

=== libffi Summary ===

# of expected passes 1444
# of unexpected failures 18
# of unsupported tests 28

which are identical to the upstream libffi 3.4.2.

Thanks.

-- 
H.J.