state of the install-bootstrap-headers patch

state of the install-bootstrap-headers patch

[Helmut Grohne]

Hi,

I'm wondering about the state of a patch that adds a flag
"install-bootstrap-headers" to the Makefile. Let me first try to
summarize the problem, then the solutions taken by various Linux
distributions and then proceed to asking how to move forward.

The <gnu/stubs.h> header defines macros of the form __stubs_$something
for each unimplemented function. It is generated at glibc install time
(not build time) by scanning the generated stubs.  Unfortunately, in a
bootstrap setting we're only installing headers and thus the stubs stuff
goes missing. This tends to result in stubs generation to be broken in
some way or another.

Thus many Linux distributions have come up with quirks to generate a
glibc header installation.

 * Debian/Ubuntu: patch
   https://sources.debian.org/src/glibc/2.27-1/debian/patches/any/local-bootstrap-headers.diff/
 * crosstool-ng: seems to be cargo culting install-bootstrap-headers=yes
   at least, but I couldn't find the actual patch.
 * Fedora/RHEL: workaround
   https://src.fedoraproject.org/rpms/glibc/blob/master/f/STAGE1-glibc-headers#_36
 * Gentoo: workaround
   https://gitweb.gentoo.org/repo/gentoo.git/tree/sys-libs/glibc/glibc-2.27-r1.ebuild#n1271
 * OpenWRT: seems to be cargo culting install-bootstrap-headers=yes at
   least, but I couldn't find the actual patch.
 * Yocto: patch
   http://git.yoctoproject.org/cgit.cgi/poky/plain/meta/recipes-core/glibc/glibc/0018-eglibc-Help-bootstrap-cross-toolchain.patch

The patches are in circulation since around 2007 and were reposted in
2012: https://sourceware.org/ml/libc-alpha/2012-03/msg00237.html The
later discussion wound down figuring whether gcc or glibc should deal
with it and it ended in some dreaming at:
https://sourceware.org/ml/libc-alpha/2012-03/msg00960.html

More than 10 years later, we still deal with this problem (revisited via
<gnu/lib-names.h>) and there still is no upstream solution, because we
cannot agree on what the solution should look like. How can we move this
forward? It causes pain on the distributors. Regardless of what a good
solution would look like, it seems that this solution is practically
being used "everywhere".  At this point I'm under the impression that
refusing the patch causes more harm than good as most users of glibc
have already agreed on how to deal with it.

Please Cc me in replies.

Helmut

Re: state of the install-bootstrap-headers patch

[DJ Delorie]

Having just gone through the Fedora RISC-V boostrap, this is fresh in my
mind (but don't think I'm a glibc Makefile expert).  Here's what I ended
up doing...

Ok, I was about to detail it, but I realized that the fedora workaround
you posted the link to... was something I wrote originally, and did for
RISC-V.  So, see that instead ;-)

Re: state of the install-bootstrap-headers patch

[Joseph Myers]

On Mon, 5 Mar 2018, Helmut Grohne wrote:

> I'm wondering about the state of a patch that adds a flag
> "install-bootstrap-headers" to the Makefile. Let me first try to
> summarize the problem, then the solutions taken by various Linux
> distributions and then proceed to asking how to move forward.

install-bootstrap-headers was part of an obsolete process for 
bootstrapping cross toolchains including glibc, which involved three GCC 
builds (a first static-only build with inhibit_libc which was used to 
configure glibc to install bootstrap headers, crt*.o and a dummy libc.so, 
which allowed a second build including shared libgcc, which could be used 
to build and install a full glibc and so allow building the third GCC 
including support for non-C languages).

The conclusion of the discussion of install-bootstrap-headers was to 
obsolete that process by enabling a two-GCC process: that is, enabling the 
initial static-only C-only GCC to build a complete glibc, with the 
installed stripped glibc binaries being byte-for-byte identical to those 
you get from a longer alternating sequence of GCC and glibc builds (in 
particular, with no dependence on the existence of libgcc_s.so at the time 
glibc is built).  As of glibc 2.16 and GCC 4.9 this is fully supported.  
(For some configurations you need the --with-glibc-version configure 
option to be passed when configuring the static-only GCC so that it knows 
what glibc version to expect to be used; in particular, this is required 
for configurations where GCC has architecture-specific SSP support.  For 
ia64 you may need GCC 5 for inhibit_libc builds to work at all.)

This modern bootstrap process is illustrated by what the 
build-many-glibcs.py script does.

> The patches are in circulation since around 2007 and were reposted in
> 2012: https://sourceware.org/ml/libc-alpha/2012-03/msg00237.html The
> later discussion wound down figuring whether gcc or glibc should deal
> with it and it ended in some dreaming at:
> https://sourceware.org/ml/libc-alpha/2012-03/msg00960.html

While not everything there (which would allow just *one* GCC build in such 
a bootstrap case) has been implemented, enough has been implemented to 
obsolete the three-GCC-build process, and thus to obsolete 
install-bootstrap-headers.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: state of the install-bootstrap-headers patch

[Joseph Myers]

On Mon, 5 Mar 2018, DJ Delorie wrote:

> Having just gone through the Fedora RISC-V boostrap, this is fresh in my
> mind (but don't think I'm a glibc Makefile expert).  Here's what I ended
> up doing...
> 
> Ok, I was about to detail it, but I realized that the fedora workaround
> you posted the link to... was something I wrote originally, and did for
> RISC-V.  So, see that instead ;-)

All those complications are completely unneccessary and have been so for 
several years.  That includes most of the configure options you pass to 
glibc's configure script (--enable-kernel=2.6.32 --disable-profile 
--without-cvs --with-elf --without-gd --disable-sanity-checks --with-tls 
--with-__thread should do nothing whatever useful in recent years for any 
supported glibc configuration) and all the *_cv_* config.cache variables 
you set (likewise obsoleted a long time ago - for example, 
libc_cv_forced_unwind and libc_cv_c_cleanup settings were obsoleted by my 
patch <https://sourceware.org/ml/libc-alpha/2012-03/msg00171.html>).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: state of the install-bootstrap-headers patch

[Andreas Schwab]

On Mär 05 2018, Joseph Myers <joseph@codesourcery.com> wrote:

> All those complications are completely unneccessary and have been so for 
> several years.  That includes most of the configure options you pass to 
> glibc's configure script (--enable-kernel=2.6.32 --disable-profile 
> --without-cvs --with-elf --without-gd --disable-sanity-checks --with-tls 
> --with-__thread should do nothing whatever useful in recent years for any 
> supported glibc configuration) and all the *_cv_* config.cache variables 
> you set (likewise obsoleted a long time ago - for example, 
> libc_cv_forced_unwind and libc_cv_c_cleanup settings were obsoleted by my 
> patch <https://sourceware.org/ml/libc-alpha/2012-03/msg00171.html>).

I can conf
Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: state of the install-bootstrap-headers patch

[Andreas Schwab]

On Mär 05 2018, Joseph Myers <joseph@codesourcery.com> wrote:

> All those complications are completely unneccessary and have been so for 
> several years.  That includes most of the configure options you pass to 
> glibc's configure script (--enable-kernel=2.6.32 --disable-profile 
> --without-cvs --with-elf --without-gd --disable-sanity-checks --with-tls 
> --with-__thread should do nothing whatever useful in recent years for any 
> supported glibc configuration) and all the *_cv_* config.cache variables 
> you set (likewise obsoleted a long time ago - for example, 
> libc_cv_forced_unwind and libc_cv_c_cleanup settings were obsoleted by my 
> patch <https://sourceware.org/ml/libc-alpha/2012-03/msg00171.html>).

When I bootstrapped openSUSE for RISC-V I didn't need any such
workarounds either.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: state of the install-bootstrap-headers patch

[DJ Delorie]

Andreas Schwab <schwab@linux-m68k.org> writes:
> When I bootstrapped openSUSE for RISC-V I didn't need any such
> workarounds either.

Oh, I didn't need the configure hacks, just the make-empty-files ones.

Re: state of the install-bootstrap-headers patch

[Helmut Grohne]

On Mon, Mar 05, 2018 at 10:00:25PM +0000, Joseph Myers wrote:
> This modern bootstrap process is illustrated by what the 
> build-many-glibcs.py script does.

I note that at least Debian still uses the longer cycle. Moving to the
shorter cycle sounds very good. It not just saves build time, but it
also removes the need for a lot of hacks to Debian's package metadata.

I'll be working to try the modern approach for Debian and thus make
Debian drop the patch I mentioned initially. I've seen
build-many-glibcs.py earlier and believe that it provides sufficient
documentation for doing so.

Thank you for your detailed reply.

Helmut

Re: state of the install-bootstrap-headers patch

[Joseph Myers]

On Mon, 5 Mar 2018, DJ Delorie wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
> > When I bootstrapped openSUSE for RISC-V I didn't need any such
> > workarounds either.
> 
> Oh, I didn't need the configure hacks, just the make-empty-files ones.

Those shouldn't be needed either.  An initial inhibit_libc GCC, built 
without any target libc headers at all, can build a complete glibc, 
without any install-headers stage needed at all.

-- 
Joseph S. Myers
joseph@codesourcery.com