[PATCH] ld: depend on libctf

[PATCH] ld: depend on libctf

[Nick Alcock]

Since ld may depend on libctf (if present), and libctf may be relinked
by the installation process, libctf must be installed before ld is,
or the relink may fail if it calls on symbols or symbol versions that do
not exist in any libctf already present on the system.  (If none is
present, the copy in the build tree will be automatically used, but
if one *is* present, it may take precedence and break things.)

(This is a maybe- dependency, so it will work even if libctf is
disabled.)

ChangeLog
2021-01-26  Nick Alcock  <nick.alcock@oracle.com>

	PR 27250
	* Makefile.def: Add install-libctf dependency to install-ld.
	* Makefile.in: Regenerated.
---
 ChangeLog    | 5 +++++
 Makefile.def | 1 +
 Makefile.in  | 1 +
 3 files changed, 7 insertions(+)

If people agree, I'll put this into master and the 2.36 branch
directly, and also get it into GCC.  It can leave you with a broken
system linker... :(

diff --git a/ChangeLog b/ChangeLog
index b6853d4abed..134df097543 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+2021-01-26  Nick Alcock  <nick.alcock@oracle.com>
+
+	* Makefile.def: Add install-libctf dependency to install-ld.
+	* Makefile.in: Regenerated.
+
 2021-01-12  Mike Frysinger  <vapier@gentoo.org>
 
 	* src-release.sh (do_proto_toplev): Rewrite indentation.
diff --git a/Makefile.def b/Makefile.def
index cc429aa8628..b45e580da5b 100644
--- a/Makefile.def
+++ b/Makefile.def
@@ -448,6 +448,7 @@ dependencies = { module=all-binutils; on=all-intl; };
 dependencies = { module=all-binutils; on=all-gas; };
 dependencies = { module=all-binutils; on=all-libctf; };
 dependencies = { module=all-ld; on=all-libctf; };
+dependencies = { module=install-ld; on=install-libctf; };
 
 // We put install-opcodes before install-binutils because the installed
 // binutils might be on PATH, and they might need the shared opcodes
diff --git a/Makefile.in b/Makefile.in
index a817b7268a0..0a64fc10e5b 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -52170,6 +52170,7 @@ all-stage3-ld: maybe-all-stage3-libctf
 all-stage4-ld: maybe-all-stage4-libctf
 all-stageprofile-ld: maybe-all-stageprofile-libctf
 all-stagefeedback-ld: maybe-all-stagefeedback-libctf
+install-ld: maybe-install-libctf
 install-binutils: maybe-install-opcodes
 install-strip-binutils: maybe-install-strip-opcodes
 install-opcodes: maybe-install-bfd
-- 
2.30.0.252.gc27e85e57d

Re: [PATCH] ld: depend on libctf

[Stephen Casner]

On Tue, 26 Jan 2021, Nick Alcock via Binutils wrote:

> Since ld may depend on libctf (if present), and libctf may be relinked
> by the installation process, libctf must be installed before ld is,
> or the relink may fail if it calls on symbols or symbol versions that do
> not exist in any libctf already present on the system.  (If none is
> present, the copy in the build tree will be automatically used, but
> if one *is* present, it may take precedence and break things.)

I need to investigate complaints about the pdp11 port in a recent
commit but make of the full binutils ran into errors shown below
related to CTF when linking objdump.  I have now cloned a fresh copy
of binutils and applied your patch, but still get these errors.  In
case it is informative, here is my configure command:

/Users/casner/src/gnu/binutils/configure --target=pdp11-aout --prefix=/usr/local

                                                        -- Steve

ld: warning: ignoring file ../libctf/.libs/libctf.a, building for macOS-x86_64 but attempting to link with file built for macOS-x86_64
Undefined symbols for architecture x86_64:
  "_ctf_archive_iter", referenced from:
      _dump_bfd in objdump.o
  "_ctf_bfdopen_ctfsect", referenced from:
      _dump_bfd in objdump.o
  "_ctf_close", referenced from:
      _dump_bfd in objdump.o
  "_ctf_dict_close", referenced from:
      _dump_bfd in objdump.o
  "_ctf_dict_open", referenced from:
      _dump_bfd in objdump.o
  "_ctf_dump", referenced from:
      _dump_ctf_archive_member in objdump.o
  "_ctf_errmsg", referenced from:
      _dump_bfd in objdump.o
      _dump_ctf_errs in objdump.o
      _dump_ctf_archive_member in objdump.o
  "_ctf_errno", referenced from:
      _dump_ctf_archive_member in objdump.o
  "_ctf_errwarning_next", referenced from:
      _dump_ctf_errs in objdump.o
  "_ctf_import", referenced from:
      _dump_ctf_archive_member in objdump.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Re: [PATCH] ld: depend on libctf

[Stephen Casner]

Looking more closely at the error message I just mentioned...

> ld: warning: ignoring file ../libctf/.libs/libctf.a, building for macOS-x86_64 but attempting to link with file built for macOS-x86_64

That doesn't make any sense because macOS-x86_64 = macOS-x86_64.  Does
building with clang no longer work?  Since the library file was
ignored, it's no surprise that there were symbols not found.

                                                        -- Steve

> Undefined symbols for architecture x86_64:
>   "_ctf_archive_iter", referenced from:
>       _dump_bfd in objdump.o
>   "_ctf_bfdopen_ctfsect", referenced from:
>       _dump_bfd in objdump.o
>   "_ctf_close", referenced from:
>       _dump_bfd in objdump.o
>   "_ctf_dict_close", referenced from:
>       _dump_bfd in objdump.o
>   "_ctf_dict_open", referenced from:
>       _dump_bfd in objdump.o
>   "_ctf_dump", referenced from:
>       _dump_ctf_archive_member in objdump.o
>   "_ctf_errmsg", referenced from:
>       _dump_bfd in objdump.o
>       _dump_ctf_errs in objdump.o
>       _dump_ctf_archive_member in objdump.o
>   "_ctf_errno", referenced from:
>       _dump_ctf_archive_member in objdump.o
>   "_ctf_errwarning_next", referenced from:
>       _dump_ctf_errs in objdump.o
>   "_ctf_import", referenced from:
>       _dump_ctf_archive_member in objdump.o
> ld: symbol(s) not found for architecture x86_64
> clang: error: linker command failed with exit code 1 (use -v to see invocation)

Re: [PATCH] ld: depend on libctf

[Stephen Casner]

It seems I am stuck, so if any of you have advice I would appreciate
it.

Switching to gcc10 did not help because the problem is ld.  I have the
MacPorts binutils package installed (2.34 is latest), but I see that
does not include ld.  The ld that I have is:

auge57> ld -v
@(#)PROGRAM:ld  PROJECT:ld64-530
BUILD 18:57:17 Dec 13 2019
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 11.0.0, (clang-1100.0.33.17) (static support for 23, runtime is 23)
TAPI support using: Apple TAPI version 11.0.0 (tapi-1100.0.11)

I have the binutils sources, of course, in order to build the pdp11
cross, but I can't build the native ld, either.  Is a binary for macOS
10.14 available somewhere?

                                                        -- Steve

On Tue, 26 Jan 2021, Stephen Casner wrote:

> Looking more closely at the error message I just mentioned...
>
> > ld: warning: ignoring file ../libctf/.libs/libctf.a, building for macOS-x86_64 but attempting to link with file built for macOS-x86_64
>
> That doesn't make any sense because macOS-x86_64 = macOS-x86_64.  Does
> building with clang no longer work?  Since the library file was
> ignored, it's no surprise that there were symbols not found.
>
>                                                         -- Steve
>
> > Undefined symbols for architecture x86_64:
> >   "_ctf_archive_iter", referenced from:
> >       _dump_bfd in objdump.o
> >   "_ctf_bfdopen_ctfsect", referenced from:
> >       _dump_bfd in objdump.o
> >   "_ctf_close", referenced from:
> >       _dump_bfd in objdump.o
> >   "_ctf_dict_close", referenced from:
> >       _dump_bfd in objdump.o
> >   "_ctf_dict_open", referenced from:
> >       _dump_bfd in objdump.o
> >   "_ctf_dump", referenced from:
> >       _dump_ctf_archive_member in objdump.o
> >   "_ctf_errmsg", referenced from:
> >       _dump_bfd in objdump.o
> >       _dump_ctf_errs in objdump.o
> >       _dump_ctf_archive_member in objdump.o
> >   "_ctf_errno", referenced from:
> >       _dump_ctf_archive_member in objdump.o
> >   "_ctf_errwarning_next", referenced from:
> >       _dump_ctf_errs in objdump.o
> >   "_ctf_import", referenced from:
> >       _dump_ctf_archive_member in objdump.o
> > ld: symbol(s) not found for architecture x86_64
> > clang: error: linker command failed with exit code 1 (use -v to see invocation)

Re: [PATCH] ld: depend on libctf

[Nick Clifton]

Hi Nick,


> ChangeLog
> 2021-01-26  Nick Alcock  <nick.alcock@oracle.com>
> 
> 	PR 27250
> 	* Makefile.def: Add install-libctf dependency to install-ld.
> 	* Makefile.in: Regenerated.

Approved - please apply.

Cheers
   Nick

Re: [PATCH] ld: depend on libctf

[Nick Alcock]

On 27 Jan 2021, Nick Clifton spake thusly:

>> ChangeLog
>> 2021-01-26  Nick Alcock  <nick.alcock@oracle.com>
>>
>> 	PR 27250
>> 	* Makefile.def: Add install-libctf dependency to install-ld.
>> 	* Makefile.in: Regenerated.
>
> Approved - please apply.

Pushed, to both 2.36 and trunk. Sorry for the brief appearance of a
phantom 'upstream/binutils-2_36-branch': I'm sure everyone can guess
just what mistake I made, for about the millionth time.

I think I'll wrap some scripting around 'git push' so it's harder to do
that... I've done it one time too many and this time on an upstream that
matters.

The change is heading up to GCC too.

Re: [PATCH] ld: depend on libctf

[Nick Alcock]

On 26 Jan 2021, Stephen Casner said:
> /Users/casner/src/gnu/binutils/configure --target=pdp11-aout --prefix=/usr/local
>
>                                                         -- Steve
>
> ld: warning: ignoring file ../libctf/.libs/libctf.a, building for macOS-x86_64 but attempting to link with file built for macOS-x86_64

I think this is your problem, whatever is causing it:

> Undefined symbols for architecture x86_64:
>   "_ctf_archive_iter", referenced from:
>       _dump_bfd in objdump.o
>   "_ctf_bfdopen_ctfsect", referenced from:
>       _dump_bfd in objdump.o

... these symbols all exist, and are in the ignored libctf.a.

Is it seriously warning it's building for arch X but attempting to link
with file built for, uh, the same arch X?

I've never seen this bug and since I don't own anything running MacOS,
and as far as I'm aware there is no way to run a MacOS system under
emulation legally, I'm not sure how I can help here other than to
vaguely point in directions you've already figured out.

I di a test build using clang+lld instead of the GNU toolchain, but
that's not it. Something else is wrong. (That test build did show up a
genuine bug, so it was a good use of time. Not one important enough to
fix on the release branch.)

libctf, gdbsupport: intl/ problems with --with-included-gettext (was Re: [PATCH] ld: depend on libctf)

[Nick Alcock]

[Redirected back to binutils@. See bug 27297 for the background.]

[In brief: please try the fix on the users/nalcock/included-gettext
 branch. It may fail because I haven't run it through my entire test
 matrix yet...]

On 2 Feb 2021, Stephen Casner outgrape:

> On Sun, 31 Jan 2021, Stephen Casner wrote:
>> I have file bug 27297 for this problem.  I hope that you or someone
>> else more familiar with the build system than I am can take it or at
>> least suggest other tests I should do.
>
> I used git bisect to find that this problem arose with your commit
> 1038406a8f6609ad0a449746da70393b0835f699 that introduced CTF_LIBADD.

Ooh thank you! Really useful debugging :)

Needless to say this behaviour change was not intended.

> Part of the change was to include an explicit reference to the
> libintl.a file (through $(LIBINTL)), whereas the references to
> libiberty and libz are with -L and -l.  If I patch Makefile.in to use
> -L../intl -lintl then that avoide the broken libctf.a in my build.

Well, that's definitely a bug (though I still don't understand what's
going on with that libctf.a.)

This reveals a hole in my testing: all my autobuilders have a system
libintl or gettext in glibc: none are using the system libintl.
Unfortunately --with-included-gettext doesn't work, with the included
libintl seeming to be incompatible with itself:

/usr/local/ctf/bin/gcc -c  -O2 -g -L/usr/local/lib -L/usr/lib   -DHAVE_CONFIG_H  -I. -I../../intl plural.c
In file included from ../../intl/plural.y:35:
../../intl/plural-exp.h:102:23: error: conflicting types for 'libintl_gettextparse'
  102 | # define PLURAL_PARSE libintl_gettextparse
      |                       ^~~~~~~~~~~~~~~~~~~~
../../intl/plural.y:40:25: note: in expansion of macro 'PLURAL_PARSE'
   40 | # define __gettextparse PLURAL_PARSE
      |                         ^~~~~~~~~~~~
plural.c:201:5: note: in expansion of macro '__gettextparse'
  201 | int __gettextparse (void);
      |     ^~~~~~~~~~~~~~
../../intl/plural-exp.h:102:23: note: previous declaration of 'libintl_gettextparse' was here
  102 | # define PLURAL_PARSE libintl_gettextparse
      |                       ^~~~~~~~~~~~~~~~~~~~

This is apparently an incompatibility with Bison 3.0: since that Bison
is seven years old now, this seems... unfortunate. glibc has fixed it by
merging libintl from gettext 0.19. GCC has fixed it differently in
commits 2ca17e0a89ff6c37e17851a5bd7b0a03ee8de535 and
437eea66a4b010d8e94aa81c2b40ccf0588e5fab from April last year.

(They seem to apply without difficulty. I have a local patch syncing
this up and will submit it soon, so nobody else need bother.)

With that in place, and --with-included-gettext, I can replicate this:

/bin/sh ./libtool  --tag=CC   --mode=link /usr/local/ctf/bin/gcc -std=gnu99 -Wall -W -Wall -Wno-narrowing -Wwrite-strings -Wmissing-format-attribute -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -pedantic -Wno-long-long  -I../../libctf/../zlib -O2 -g -L/usr/local/lib -L/usr/lib   -version-info 0:0:0  -Wl,--version-script='../../libctf/libctf.ver' -Wl,-rpath=/usr/local/ctf/lib -o libctf-nobfd.la -rpath /usr/local/ctf/lib libctf_nobfd_la-ctf-archive.lo libctf_nobfd_la-ctf-dump.lo libctf_nobfd_la-ctf-create.lo libctf_nobfd_la-ctf-decl.lo libctf_nobfd_la-ctf-error.lo libctf_nobfd_la-ctf-hash.lo libctf_nobfd_la-ctf-labels.lo libctf_nobfd_la-ctf-dedup.lo libctf_nobfd_la-ctf-link.lo libctf_nobfd_la-ctf-lookup.lo libctf_nobfd_la-ctf-open.lo libctf_nobfd_la-ctf-sha1.lo libctf_nobfd_la-ctf-string.lo libctf_nobfd_la-ctf-subr.lo libctf_nobfd_la-ctf-types.lo libctf_nobfd_la-ctf-util.lo  -L/usr/home/nix/binutils/build/libctf/../libiberty -liberty ./../intl/libintl.a -liconv -L./../zlib -lz 

libtool: link: ar  cru .libs/libctf-nobfd.a ./../intl/libintl.a  libctf_nobfd_la-ctf-archive.o libctf_nobfd_la-ctf-dump.o libctf_nobfd_la-ctf-create.o libctf_nobfd_la-ctf-decl.o libctf_nobfd_la-ctf-error.o libctf_nobfd_la-ctf-hash.o libctf_nobfd_la-ctf-labels.o libctf_nobfd_la-ctf-dedup.o libctf_nobfd_la-ctf-link.o libctf_nobfd_la-ctf-lookup.o libctf_nobfd_la-ctf-open.o libctf_nobfd_la-ctf-sha1.o libctf_nobfd_la-ctf-string.o libctf_nobfd_la-ctf-subr.o libctf_nobfd_la-ctf-types.o libctf_nobfd_la-ctf-util.o

[nix@freebsd /usr/home/nix/binutils/build]$ ar t libctf/.libs/libctf-nobfd.a | less
libintl.a
libctf_nobfd_la-ctf-archive.o
(...)

Observe the direct reference to ./../intl/libintl.a, rather than
-L../intl -lintl. I bet that's the root of this. (It doesn't break any
platform I've tried it on, but it's still pretty clearly wrong.)

> I don't know how to get that form automatically pulled into
> Makefile.am.  Would that be a proper fix for this problem?

Even if this is fixed, linking with --enable-shared fails because
libintl is not built with -fPIC: even if that is fixed the obvious way,
gdbsupport breaks because it links shared libraries that need gettext
even when --disable-shared is specified!


I have a possible fix for all of this against a slightly old trunk on
the users/nalcock/included-gettext branch: could you try it out? (It's
several commits long, so pointing you at the git branch seems the
easiest way to have you test it.)

(The gdbsupport part, at least, needs reporting as a bug, but I can't do
that yet and I don't recommend anyone else try: when I tried, my bz
account was reported for 'inappropriate content' and locked for
spamming. Yaaaay. Reported to overseers@... I suppose that's the danger
of distributed operations like gdbserver enables: it's intrinsically
racy!)

Re: libctf, gdbsupport: intl/ problems with --with-included-gettext (was Re: [PATCH] ld: depend on libctf)

[Stephen Casner]

On Tue, 2 Feb 2021, Nick Alcock wrote:
>
> I have a possible fix for all of this against a slightly old trunk on
> the users/nalcock/included-gettext branch: could you try it out? (It's
> several commits long, so pointing you at the git branch seems the
> easiest way to have you test it.)

With the following configure:

auge6> ~/src/gnu/binutils-gdb/configure --prefix=/usr/local --with-gmp=/opt/local

It builds through gdbsupport, but I have an unrelated problem with gdb
configure:

checking for libgmp... no
configure: error: GMP is missing or unusable

This even though I have the --with-gmp and /usr/local/lib/libgmp.*
exist.  I need to investigate that further.

So my test run has not verified that everything links properly with
all the libraries.

Also presumably unrelated, there are these warnings:

/opt/local/bin/ranlib: file: .libs/libbfd.a(bfdwin.o) has no symbols
/opt/local/bin/ranlib: file: libgdbsupport.a(gdb_wait.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(count-one-bits.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(fd-hook.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(getcwd-lgpl.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(getprogname.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(localtime-buffer.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(math.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(stat-time.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(sys_socket.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(threadlib.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(unistd.o) has no symbols
/opt/local/bin/ranlib: file: libgnu.a(wctype-h.o) has no symbols
/opt/local/bin/ranlib: file: libintl.a(osdep.o) has no symbols

I've seen some discussions on this list about having no symbols, but
my build is using the system ld.

                                                        -- Steve

Re: libctf, gdbsupport: intl/ problems with --with-included-gettext (was Re: [PATCH] ld: depend on libctf)

[Nick Alcock]

On 2 Feb 2021, Stephen Casner outgrape:

> On Tue, 2 Feb 2021, Nick Alcock wrote:
>>
>> I have a possible fix for all of this against a slightly old trunk on
>> the users/nalcock/included-gettext branch: could you try it out? (It's
>> several commits long, so pointing you at the git branch seems the
>> easiest way to have you test it.)
>
> With the following configure:
>
> auge6> ~/src/gnu/binutils-gdb/configure --prefix=/usr/local --with-gmp=/opt/local
>
> It builds through gdbsupport, but I have an unrelated problem with gdb
> configure:
>
> checking for libgmp... no
> configure: error: GMP is missing or unusable

Definitely unrelated, I'd say.

> So my test run has not verified that everything links properly with
> all the libraries.

The link failures I saw with --with-included-gettext lay in libctf, ld,
and gdbserver/libinproctrace.so. GDB itself seemed happy.

> Also presumably unrelated, there are these warnings:
>
> /opt/local/bin/ranlib: file: .libs/libbfd.a(bfdwin.o) has no symbols
> /opt/local/bin/ranlib: file: libgdbsupport.a(gdb_wait.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(count-one-bits.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(fd-hook.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(getcwd-lgpl.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(getprogname.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(localtime-buffer.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(math.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(stat-time.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(sys_socket.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(threadlib.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(unistd.o) has no symbols
> /opt/local/bin/ranlib: file: libgnu.a(wctype-h.o) has no symbols
> /opt/local/bin/ranlib: file: libintl.a(osdep.o) has no symbols
> 
> I've seen some discussions on this list about having no symbols, but
> my build is using the system ld.

Yeah, I don't think this could possibly be related, though perhaps it is
exacerbated a bit: I didn't touch the gnulib or libbfd side of thing at
all.

I'll start throwing this through testing, but I don't think I'm going to
try to get it upstream (or into the branch) until after I've got my
current patch series in, if that's OK? It probably needs more review
from the GDB side and, uh... whoever the heck reviews changes to intl/ :)

Re: libctf, gdbsupport: intl/ problems with --with-included-gettext (was Re: [PATCH] ld: depend on libctf)

[Stephen Casner]

On Tue, 2 Feb 2021, Nick Alcock wrote:

> I'll start throwing this through testing, but I don't think I'm going to
> try to get it upstream (or into the branch) until after I've got my
> current patch series in, if that's OK? It probably needs more review
> from the GDB side and, uh... whoever the heck reviews changes to intl/ :)

Fine with me.  Thanks for picking this up.

                                                        -- Steve