Re: Question about autoreconf to regenerate configuration files

[Mike Frysinger <vapier@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 7542 bytes --]

On 22 Jan 2022 22:20, R. Diez wrote:
> Newlib's build system has been a major pain

it's no more painful than any other autotool based system.  if you mean
installing the specific version of autotools on your system is a major
issue, i don't think that's true.  it's pretty easy to download locally
each version and run them directly.

> > [...]
> > autotools (autoreconf really) doesn't run in parallel, so every subdir
> > with a configure script needs a separate serialized run of all the tools.
> > newlib has many many of these (arguably, too many).
> > 
> > on my quad core 4.2GHz AMD that is otherwise idle ...
> > 
> > $ time (cd newlib && autoreconf)
> > real    5m22.170s
> > user    3m13.709s
> > sys     0m12.332s
> 
> My system is not really a very fast machine for development purposes:
> - A laptop with an Intel Core i5-8265U CPU @ 1.60GHz (4 cores + hyperthreading)
> - 1TB consumer-grade SSD Crucial MX500
> - Ubuntu 20.04
> My result is:
> 
> $ ( cd newlib && time autoreconf --force --verbose )
> real    3m47,685s
> user    1m48,533s
> sys     0m10,338s

i was a bit lazy and didn't reset my disk caches, clear the autom4te.cache
trees, and make sure the set of system packages installing into the common
m4 tree (/usr/share/aclocal) was minimized.  so i would expect our times to
vary a bit.  i think the scale is still relevant though -- we aren't talking
O(<10 seconds) here, we're talking O(minutes).

> That's actually a rather long time. But if I understand correctly, your target
> is to have just 2 'configure' scripts for newlib and libgloss, which means that
> the Autoconf regeneration time is going to get cut drastically. Is that right?

the autoconf part will, but autoreconf is doing more than that.  i have not
split apart the m4 processing (aclocal), aux file processing, the libtoool,
or the automake steps.

i know the names are similar ("autoconf" and "autoreconf") and can be a bit
confusing, but while "autoconf" does just one thing (turn configure.ac into
configure), "autoreconf" drives all possible autotools.

> Is it actually a problem that regenerating the Autoconf files in Newlib takes 5 minutes?

yes

> Is the Newlib project doing Continuous Integration?

the project itself is not, but there are a lot of downstream folks who are.
i.e. your redhats and your windrivers and such (i'm aware that the industry
has gone through a lot of shuffling/acquiring and me quoting companies that
have been fully bought out dates me -- the work is still being done somewhere).

> If so, it should actually be doing the Autoconf regeneration anyway, in order to test it as well.

sure, iff the autotools are touched.  if they aren't, and they often aren't
by most devs, then it's a waste of time.

> In any case, would a 5-minute delay there be an issue?

i think you're in the old world thinking of "CI is that big slow thing that
runs periodically and sends me feedback that i look at hours or days later".
we have systems now that can provide much faster feedback.  presubmit checks
can get down to <minute levels.

plus, telling your developers that build-system-generation takes 80% of their
time bofore they get to the 20% that actually matters to them is a pretty hard
sell.

yes, i grok that this step doesn't always need to be run.  e.g. if you do a
`git pull` and get updated tools, you'll need to refresh, but if you're just
hacking on source files, you won't.  keep in mind that toolchain devs often
deal with regressions which involve git bisect, and if that requires ~minutes
everytime you checkout a diff tree state, as well as possibly needing to pick
a diff version of autotools if the bisect spans multiple releases, that's quite
a lot of pain.

committing the generated files, as ugly as it is, avoids all of this.

> The other way to build such a cross-compiler toolchain is to merge Newlib's sources with GCC's etc.
> That is called a "combined tree" and is mentioned here:
> 
> https://gcc.gnu.org/wiki/Building_Cross_Toolchains_with_gcc
> 
> The instructions on that page appear outdated and do not seem complete.

they do look outdated.  i don't use this flow myself as i rarely hack on gcc.
i get most of my cross-compilers from gcc releases.  i develop binutils, gdb,
newlib, and the sim myself.

> If you are keeping the Newlib build system compatible with GCC's etc,
> you must be testing with some commands, or maybe a script, that merges the trees.
> Could you share those steps?

i just manually symlink or bind mount the project subdirs i care about.

> The instructions on the page I mentioned above talk about GCC's files overwriting any files in Newlib etc.
> I wonder at what level that is supposed to happen. For example, gcc-10.3.0 has a top-level 'configure' script.
> Is that supposed to overwrite Newlib's top-level 'configure' script?

there is only one set of top-level files.  they get manually synced from gcc
to the other projects by devs who notice & need the fixes.

> In the combined tree, GCC must know somehow that it should include the 'newlib' subdirectory.
> Is that what GCC's 'configure' option '--with-newlib' is supposed to do?
> I could not find any information about "--with-newlib" in the GCC documentation.

that's because the top-level scripts contain a superset of options, and many
of them do dynamic probing of subdirectories.  --with-newlib is an option in
every toolchain project -- grab a binutils, gdb, gcc, or newlib release, and
you'll see it in all of them.  it just doesn't make sense in most.

think of the top-level script as a "looks for projects to build, and then passes
all arguments down to them".  it doesn't really do much else.  it's a glorified
mux and is full of shell `case` statements as a result.

if you look at the top-level configure.ac, you'll see a block near the top like:
### To add a new directory to the tree, first choose whether it is a target
### or a host dependent tool.  Then put it into the appropriate list
### (library or tools, host or target), doing a dependency sort.

after that you can see all of the projects that are supported in a combined tree.

> To what extent is build system compatibility in this GCC ecosystem a problem?
> Newlib has had very outdated Autoconf files for quite some time and I presume
> that it has been working with many different GCC versions over the years.

the contract at build time is `./configure`, and even if newlib had its script
generated with an old version of autoconf, it doesn't really change the API.
the top-level Makefile knows to `cd newlib && ./configure ...` and that will
work regardless of the autoconf version.

> I do not suppose that we maintain a table of build system version compatibility
> between Newlib and GCC releases, do we?

probably not.  i don't think anyone maintains a large matrix of binutils/gdb/gcc
and the newlib/glibc libraries.  usually the answer is "pick versions that are
released around the sametime", and the larger you try to skew that window, the
more "it's your problem" to make them work.

> My guess is that GCC's build system is calling 'configure', 'make' etc. inside
> newlib/, and it does not really matter if the versions are a little different.
> Or is a lot of the build system really shared?

the use of `./configure ...` and `make ...` are the most important.  the multilib
multiplexing logic with libraries makes things a bit messier too.
-mike

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]