Re: Question about autoreconf to regenerate configuration files

[Joel Sherrill <joel@rtems.org>]

On Sat, Jan 22, 2022, 3:20 PM R. Diez <rdiezmail-newlib@yahoo.de> wrote:

> Hallo Mike:
>
> Newlib's build system has been a major pain and I would like to take this
> opportunity to learn more about it, and if that also helps with the other
> toolchain components, it's a plus point.
>
>
> > [...]
> > autotools (autoreconf really) doesn't run in parallel, so every subdir
> > with a configure script needs a separate serialized run of all the tools.
> > newlib has many many of these (arguably, too many).
> >
> > on my quad core 4.2GHz AMD that is otherwise idle ...
> >
> > $ time (cd newlib && autoreconf)
> > real    5m22.170s
> > user    3m13.709s
> > sys     0m12.332s
>
> My system is not really a very fast machine for development purposes:
> - A laptop with an Intel Core i5-8265U CPU @ 1.60GHz (4 cores +
> hyperthreading)
> - 1TB consumer-grade SSD Crucial MX500
> - Ubuntu 20.04
> My result is:
>
> $ ( cd newlib && time autoreconf --force --verbose )
> real    3m47,685s
> user    1m48,533s
> sys     0m10,338s
>
> That's actually a rather long time. But if I understand correctly, your
> target is to have just 2 'configure' scripts for newlib and libgloss, which
> means that the Autoconf regeneration time is going to get cut drastically.
> Is that right?
>
> Is it actually a problem that regenerating the Autoconf files in Newlib
> takes 5 minutes?
>
> Is the Newlib project doing Continuous Integration? If so, it should
> actually be doing the Autoconf regeneration anyway, in order to test it as
> well. In any case, would a 5-minute delay there be an issue?
>

The RTEMS project checks three times a day and builds a target for
Coverity.

We build full tool chains mates to three  branches on about 8 OSes once a
week. With just under 20 targets, 3 branches and 8 VMs for the OSes, that's
240 builds a week for test builds. All are.on the same physical computer.
We could be looking at 12 to 20 hours more of build time a week.

Would reducing the number of Makefile.am files help newlib? That worked for
RTEMS. Also allowed make to see more parallelism. Portable libc, portable
libc, and one per machine or sys directory. Just thinking out loud.


> This Autoconf-related delay would only apply to sources directly obtained
> from the Git repository. The Newlib tarball releases would already include
> the Autoconf files, so no delay there.
>
> If somebody, like RTEMS, is tracking Newlib's Git repository directly,
> that's their choice. But even then, they have options. If they distribute a
> release with patches etc, they can regenerate the Autoconf files before
> packaging Newlib for their own release. If the problem lies in the
> Continuous Integration system, you can cache the Autoconf files: if last
> time you checked out the same Git commit hash, then you do not need to
> regenerate the Autotools files again.
>

Yes that is possible. Not sure how this would work on our system but
possible.

>
>
> > again, this isn't "just newlib".  newlib is part of the historically
> combined
> > toolchain tree/ecosystem.  that means you can take binutils, gdb, gcc,
> newlib,
> > libgloss, cgen, sim, zlib, etc... and have a single monolithic source
> tree and
> > build them all at once.  the projects have separated a little bit in
> that they
> > have diff git repos, but the top-level dir and a few subdirs are still
> shared,
> > and some folks still hand merge them.  newlib is part of that ecosystem
> and as
> > such, follows its conventions.  changing newlib behavior would have a
> ripple
> > effect and is why consensus across all of them is desirable.  although
> usually
> > if you can convince gcc to change, the rest will follow to keep things
> simple.
>
> OK, let me try to understand the details here.
>
> I have been using the following makefile for years to build a
> cross-compiler GCC toolchain:
>
> https://github.com/rdiez/JtagDue/blob/master/Toolchain/Makefile
>
> That makefile builds GCC in 2 phases: a first, temporary GCC without a C
> library, and the final GCC with Newlib.
>
> With that makefile, Newlib is configured, built and installed separately
> from GCC, so Newlib's build system does not matter at all in this scenario.
>
> I think that is the same strategy as the OpenWrt toolchain makefile:
>    https://git.openwrt.org/
>    toolchain/Makefile
> This way, you can use another libc. I believe that OpenWrt tends to use
> Musl instead of glibc.
>
>
> The other way to build such a cross-compiler toolchain is to merge
> Newlib's sources with GCC's etc. That is called a "combined tree" and is
> mentioned here:
>
> https://gcc.gnu.org/wiki/Building_Cross_Toolchains_with_gcc
>
> The instructions on that page appear outdated and do not seem complete.
>

We do thia one tree style.

>
> If you are keeping the Newlib build system compatible with GCC's etc, you
> must be testing with some commands, or maybe a script, that merges the
> trees. Could you share those steps? Alternatively, do you know of a web
> page that describes the steps accurately?
>
> The instructions on the page I mentioned above talk about GCC's files
> overwriting any files in Newlib etc. I wonder at what level that is
> supposed to happen. For example, gcc-10.3.0 has a top-level 'configure'
> script. Is that supposed to overwrite Newlib's top-level 'configure' script?
>

We move or symlink the newlib/ directory under the top GCC directory.

>
> In the combined tree, GCC must know somehow that it should include the
> 'newlib' subdirectory. Is that what GCC's 'configure' option
> '--with-newlib' is supposed to do? I could not find any information about
> "--with-newlib" in the GCC documentation.
>
>
> To what extent is build system compatibility in this GCC ecosystem a
> problem? Newlib has had very outdated Autoconf files for quite some time
> and I presume that it has been working with many different GCC versions
> over the years. I do not suppose that we maintain a table of build system
> version compatibility between Newlib and GCC releases, do we?
>

We've been doing it this way since the early 90s. No noticeable difference
in procedure since the old Cygnus one tree instructions.

>
> My guess is that GCC's build system is calling 'configure', 'make' etc.
> inside newlib/, and it does not really matter if the versions are a little
> different. Or is a lot of the build system really shared?
>

Based on how we always build, I'd say it is just invocation.

--joel

>
> --
> rdiez
>