From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.gentoo.org (smtp.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) by sourceware.org (Postfix) with ESMTP id 594203858D37 for ; Sun, 23 Jan 2022 07:29:01 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 594203858D37 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gentoo.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gentoo.org Received: by smtp.gentoo.org (Postfix, from userid 559) id B2E08343124; Sun, 23 Jan 2022 07:29:00 +0000 (UTC) Date: Sun, 23 Jan 2022 02:29:03 -0500 From: Mike Frysinger To: "R. Diez" Cc: Newlib , Matthew Joyce , joel@rtems.org Subject: Re: Question about autoreconf to regenerate configuration files Message-ID: Mail-Followup-To: "R. Diez" , Newlib , Matthew Joyce , joel@rtems.org References: <26fff1a3-7743-e0ce-f7a6-70b39a030118@embedded-brains.de> <41deefa7-d1ae-18f2-6a8e-25e002d85ed6@yahoo.de> <2bc01390-3a5b-fd2d-822c-c4cd9a96f744@yahoo.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="739BjBCnFp4TlAnt" Content-Disposition: inline In-Reply-To: <2bc01390-3a5b-fd2d-822c-c4cd9a96f744@yahoo.de> X-Spam-Status: No, score=-5.6 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_SHORT, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: newlib@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Newlib mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Jan 2022 07:29:05 -0000 --739BjBCnFp4TlAnt Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 22 Jan 2022 22:20, R. Diez wrote: > Newlib's build system has been a major pain it's no more painful than any other autotool based system. if you mean installing the specific version of autotools on your system is a major issue, i don't think that's true. it's pretty easy to download locally each version and run them directly. > > [...] > > autotools (autoreconf really) doesn't run in parallel, so every subdir > > with a configure script needs a separate serialized run of all the tool= s. > > newlib has many many of these (arguably, too many). > >=20 > > on my quad core 4.2GHz AMD that is otherwise idle ... > >=20 > > $ time (cd newlib && autoreconf) > > real 5m22.170s > > user 3m13.709s > > sys 0m12.332s >=20 > My system is not really a very fast machine for development purposes: > - A laptop with an Intel Core i5-8265U CPU @ 1.60GHz (4 cores + hyperthre= ading) > - 1TB consumer-grade SSD Crucial MX500 > - Ubuntu 20.04 > My result is: >=20 > $ ( cd newlib && time autoreconf --force --verbose ) > real 3m47,685s > user 1m48,533s > sys 0m10,338s i was a bit lazy and didn't reset my disk caches, clear the autom4te.cache trees, and make sure the set of system packages installing into the common m4 tree (/usr/share/aclocal) was minimized. so i would expect our times to vary a bit. i think the scale is still relevant though -- we aren't talking O(<10 seconds) here, we're talking O(minutes). > That's actually a rather long time. But if I understand correctly, your t= arget > is to have just 2 'configure' scripts for newlib and libgloss, which mean= s that > the Autoconf regeneration time is going to get cut drastically. Is that r= ight? the autoconf part will, but autoreconf is doing more than that. i have not split apart the m4 processing (aclocal), aux file processing, the libtoool, or the automake steps. i know the names are similar ("autoconf" and "autoreconf") and can be a bit confusing, but while "autoconf" does just one thing (turn configure.ac into configure), "autoreconf" drives all possible autotools. > Is it actually a problem that regenerating the Autoconf files in Newlib t= akes 5 minutes? yes > Is the Newlib project doing Continuous Integration? the project itself is not, but there are a lot of downstream folks who are. i.e. your redhats and your windrivers and such (i'm aware that the industry has gone through a lot of shuffling/acquiring and me quoting companies that have been fully bought out dates me -- the work is still being done somewhe= re). > If so, it should actually be doing the Autoconf regeneration anyway, in o= rder to test it as well. sure, iff the autotools are touched. if they aren't, and they often aren't by most devs, then it's a waste of time. > In any case, would a 5-minute delay there be an issue? i think you're in the old world thinking of "CI is that big slow thing that runs periodically and sends me feedback that i look at hours or days later". we have systems now that can provide much faster feedback. presubmit checks can get down to The other way to build such a cross-compiler toolchain is to merge Newlib= 's sources with GCC's etc. > That is called a "combined tree" and is mentioned here: >=20 > https://gcc.gnu.org/wiki/Building_Cross_Toolchains_with_gcc >=20 > The instructions on that page appear outdated and do not seem complete. they do look outdated. i don't use this flow myself as i rarely hack on gc= c. i get most of my cross-compilers from gcc releases. i develop binutils, gd= b, newlib, and the sim myself. > If you are keeping the Newlib build system compatible with GCC's etc, > you must be testing with some commands, or maybe a script, that merges th= e trees. > Could you share those steps? i just manually symlink or bind mount the project subdirs i care about. > The instructions on the page I mentioned above talk about GCC's files ove= rwriting any files in Newlib etc. > I wonder at what level that is supposed to happen. For example, gcc-10.3.= 0 has a top-level 'configure' script. > Is that supposed to overwrite Newlib's top-level 'configure' script? there is only one set of top-level files. they get manually synced from gcc to the other projects by devs who notice & need the fixes. > In the combined tree, GCC must know somehow that it should include the 'n= ewlib' subdirectory. > Is that what GCC's 'configure' option '--with-newlib' is supposed to do? > I could not find any information about "--with-newlib" in the GCC documen= tation. that's because the top-level scripts contain a superset of options, and many of them do dynamic probing of subdirectories. --with-newlib is an option in every toolchain project -- grab a binutils, gdb, gcc, or newlib release, and you'll see it in all of them. it just doesn't make sense in most. think of the top-level script as a "looks for projects to build, and then p= asses all arguments down to them". it doesn't really do much else. it's a glori= fied mux and is full of shell `case` statements as a result. if you look at the top-level configure.ac, you'll see a block near the top = like: ### To add a new directory to the tree, first choose whether it is a target ### or a host dependent tool. Then put it into the appropriate list ### (library or tools, host or target), doing a dependency sort. after that you can see all of the projects that are supported in a combined= tree. > To what extent is build system compatibility in this GCC ecosystem a prob= lem? > Newlib has had very outdated Autoconf files for quite some time and I pre= sume > that it has been working with many different GCC versions over the years. the contract at build time is `./configure`, and even if newlib had its scr= ipt generated with an old version of autoconf, it doesn't really change the API. the top-level Makefile knows to `cd newlib && ./configure ...` and that will work regardless of the autoconf version. > I do not suppose that we maintain a table of build system version compati= bility > between Newlib and GCC releases, do we? probably not. i don't think anyone maintains a large matrix of binutils/gd= b/gcc and the newlib/glibc libraries. usually the answer is "pick versions that = are released around the sametime", and the larger you try to skew that window, = the more "it's your problem" to make them work. > My guess is that GCC's build system is calling 'configure', 'make' etc. i= nside > newlib/, and it does not really matter if the versions are a little diffe= rent. > Or is a lot of the build system really shared? the use of `./configure ...` and `make ...` are the most important. the mu= ltilib multiplexing logic with libraries makes things a bit messier too. -mike --739BjBCnFp4TlAnt Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEuQK1JxMl+JKsJRrUQWM7n+g39YEFAmHtA78ACgkQQWM7n+g3 9YFnhQ/9EMM8h8/JZdLjFS7c0LWPSLeLIWg+zdb0sOI+17J9KpLeVHKvjnnNdMq2 KEzYr5jbO2hpOfFpoAUFJqQPdoSj/QJb0e/7I5hd5biMYHn9pHWkAD94nKbDTk8W 2p3U4KGzGL0j9l/st70gNc4dWldAaJhJSGX40AJsNxLxAfigA1oqvXdCPSgWkmPr bfGSwucsYUpvYhflJJ3yh7spfYrdmc4sFOwQMOfqeUs9EBh26i9CqeONTlyWU+6N K+qY2230aSjdhXKgfow4obEl1TjqLM7wBca2gaQOpMxa7/rXfjOXPORSueFvmAd9 CSWBNBXdTHw1EF3xDacJLuEUXKgQ44YgQwGVdDdthJMmhI2dFSpt8NxfCvwYwfqn EuiRb6wEr4p1hkxF3I6ud1kJMkJt5kvZGPRmWllrS1ckxqmidXjxXjSg8qHBZi/E o1dLncLbQiCg8h7lXJurfh88xGwsp7POhijnhU7dbwuG+sy1ZF4IuWjqDMQvjB7k lJ7qUZZ2npozeOtqxJ0LVmU4ww7gjit0QfzURsUGXdiRUoLvb3PPCRuYejOwb/Be 5VC7BnhBJHMwruUPx44Re5ndyG4KHVceYTO8y/Dx1tUnbINKgUjBRRwvTkDUkUye +Y6ebIHVBcw31TNojvEQWNtaT8nuFwS3doy809kz+JBtlGIgzD4= =T32A -----END PGP SIGNATURE----- --739BjBCnFp4TlAnt--