From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by sourceware.org (Postfix) with ESMTPS id 887FE3858289 for ; Tue, 19 Mar 2024 17:12:01 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 887FE3858289 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 887FE3858289 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::534 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710868324; cv=none; b=NQowOvnFGr36CNAGu+aVsK6oM+WqAJkKUiivMJ9YcJIIG9uNQxbTMjoctzv81Eh85mSX2r42bSOJdSo/J++eJ6WXDWySLt8u0YQxT6g7wRuqMk3p9Se99ZuBsd0zB5gBB6S/DGyL/OhGoVjrDSaqJGEuNbZf1YSKq1FDdB0K8NM= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710868324; c=relaxed/simple; bh=1Irdl2xbkCD5vKjnop/kGfhukEkkysKbDEVXM9PwNqw=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=JdkwLzugAHDjTbCbW3dbZT1atLvmGAQFgZv65j9dr2YRQd6QNihjP6zrQFYQ3tNYWSHkycz/fEL0yXsQJZzrY5wM5PDNKINVLGTAfij5BkWVeuyBp7GPcrMBtVG0olLGedYQhh8KcLm8yR3kt1XEvNecQLLKNAGlbo8id8l8wCs= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-56890b533aaso6609484a12.3 for ; Tue, 19 Mar 2024 10:12:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1710868320; x=1711473120; darn=sourceware.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=NILCR1aPLJTPDzGn6hGYjQwlu3CSBdSRaJlHStnSy08=; b=SzYFKi5WmwxItc01Zs0Tq2V1YrQBa8wRRQtlOXqj0NYQZGT2S+r1yJxzMFmGHez0wD E/UskwrYpFym+cp70UZKcGGW8vmmE2BtJMnBlE/Mn7kyVwlpRfE51+LJJ/ZwfdELk0cA C9Ky5mJLgM5eXgUqDqpkCT61KEDaSdFYkH3/zasi7pU5qbT2k2SZL1kIobwzbVC4MDV/ 09h1MEZcGgHuAeLH+OuCUEzpii/zzO6ifaV4/JT0NbxIDBh5FMPxrG6tT38OkgLFV+Ia whcFjrbLKaQhhxgHKXIKjayBmMKVSGKlhoQpMwonOWZ4F4wpfraz7dGrBDglHNapL9NM dCIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710868320; x=1711473120; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NILCR1aPLJTPDzGn6hGYjQwlu3CSBdSRaJlHStnSy08=; b=c1wfmkGQrQ7Z7DSIkgSn3vqVPrrvanyrNfttUsE3j+nHLM4QC4MEMMl5BtNG8P8hzp Sr6hldmRYoVkw8mRMa40t8P6D4lHdGU/90kOl4Lorxqu1sVLrPyXTpF4QGZUL/v1zkUS ug+L/QcLH2UKI5NcwikFlJjP/ntEKegOI6+IjCe5x1sTZGIcgyBmOr+ZjM42G3Swk1iV 7rWvCcPvym+SjVXeKuYwmWSiYYcAJV3pMkqytl3FaT1Vn3dFonOe6RgXdi7hU7JQO60S ZpKGNoT5ka/a+qxmdL0ww1R/CNtGf7ZBFJ5GNgsw8Y+J+EHNd9Vd2AUyc/d239rBomyp fn6A== X-Forwarded-Encrypted: i=1; AJvYcCXUD19Gm5cRhaYbmD9lviqYrXc2twi2RKLI+Gl6RyODTrembzuIN09NkUvPC0r1t2kN8saz5bJM4D+XO0Xe9FZgB4o= X-Gm-Message-State: AOJu0YxK+l9s8WF8D1JLKuF4wzNdRSkpYb4NzXCtpN4MHWeR/RzPCnTo 43v0h3EFI3S8rWzQrTUO5PrW459P2BimMIdn3XKD4mp7ffmKC0N5wR/VReNTvyTJPVFgSvr0qNI erH/Hev64IN3v3u9UouFK7hf+bvVSnSddxj54MA== X-Google-Smtp-Source: AGHT+IGgw3TPSihQgdwUF7s/79+QpROhY2+H2VCG+xFXq/c1R40mFOevc4N4Qp1nZ8yjZC0QtReT6wJxl2KiH/WdZAM= X-Received: by 2002:a05:6402:3884:b0:568:d5e7:37a1 with SMTP id fd4-20020a056402388400b00568d5e737a1mr2026951edb.36.1710868320051; Tue, 19 Mar 2024 10:12:00 -0700 (PDT) MIME-Version: 1.0 References: <20240313080237.1143034-1-christophe.lyon@linaro.org> <1eb529f2-3842-4090-a8e2-f713a28f2394@simark.ca> In-Reply-To: From: Christophe Lyon Date: Tue, 19 Mar 2024 18:11:49 +0100 Message-ID: Subject: Re: [RFC] add regenerate Makefile target To: Simon Marchi Cc: binutils@sourceware.org, gdb@sourceware.org, gcc@gcc.gnu.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi, On Mon, 18 Mar 2024 at 18:25, Christophe Lyon wrote: > > On Sat, 16 Mar 2024 at 18:16, Simon Marchi wrote: > > > > > > > > On 2024-03-15 04:50, Christophe Lyon via Gdb wrote: > > > On Thu, 14 Mar 2024 at 19:10, Simon Marchi wrote: > > >> My first thought it: why is it a Makefile target, instead of some script > > >> on the side (like autoregen.sh). It would be nice / useful to be > > >> able to it without configuring / building anything. For instance, the > > >> autoregen buildbot job could run it without configuring anything. > > >> Ideally, the buildbot wouldn't maintain its own autoregen.py file on the > > >> side, it would just use whatever is in the repo. > > > > > > Firstly because of what you mention later: some regeneration steps > > > require building host tools first, like the XXX-gen in opcodes. > > > > "build" and not "host", I think? > > yes, sorry > > > > Since the existing Makefiles already contain the rules to autoregen > > > all these files, it seemed natural to me to reuse them, to avoid > > > reinventing the wheel with the risk of introducing new bugs. > > > > I understand. Although one advantage of moving the actual code out of > > the Makefile (even if there's still a Makefile rule calling the external > > script), is that it's much easier to maintain. Editors are much more > > useful when editing a standalone shell script than editing shell code in > > a Makefile target. It doesn't have to be this big one liner if you want > > to use variables, you don't need to escape $, you can run it through > > linters, you can call it by hand, etc. This is what I did here, for > > instance: > > > > https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=f39632d9579d3c97f1e50a728efed3c5409747d2 > > > > So I think there's value in any case of moving the regeneration logic > > out of the Makefiles per se. > > > In this case, the generation rules look simple enough indeed. > But as mentioned elsewhere in the thread, there are more complex > cases, which involve building helper tools, which have dependencies on > bfd and libiberty for instance. I'm not sure that's easily/naturally > scriptable? > There's also 'chew' in bfd/ > > > > This involves changes in places where I've never looked at before, so > > > I'd rather reuse as much existing support as possible. > > > > > > For instance, there are the generators in opcodes/, but also things in > > > sim/, bfd/, updates to the docs and potfiles. In gcc, there's also > > > something "unusual" in fixincludes/ and libgfortran/ > > > > > > In fact, I considered also including 'configure', 'Makefile.in', > > > etc... in the 'regenerate' target, it does not seem natural to me to > > > invoke a script on the side, where you have to replicate the behaviour > > > of existing Makefiles, possibly getting out-of-sync when someone > > > forgets to update either Makefile or autoregen.py. > > > > I'm not sure I follow. Are you referring to the rules that automake > > automatically puts to re-generate Makefile.in and others when > > Makefile.am has changed? Your regenerate target would depend on those > > builtin rules? > Yes, "regenerate" would include "configure, Makeifile.in, configh" > (as/if needed) in its list of dependencies. > > > > > Let's say my generate-autostuff.sh script does: > > > > aclocal --some-flags > > automake --some-other-flags > > autoconf --some-other-other-flags > > > > And the checked-in Makefile.in is regenerated based on that. Wouldn't > > the built-in rules just call aclocal/automake/autoconf with those same > > flags? I don't see why they would get out of sync. > Well the rule to regenerate Makefile.in (eg in in opcodes/) is a bit > more complex > than just calling automake. IIUC it calls automake --foreign it any of > *.m4 file from $(am__configure_deps) that is newer than Makefile.in > (with an early exit in the loop), does nothing if Makefile.am or > doc/local.mk are newer than Makefile.in, and then calls 'automake > --foreign Makefile' > > I've never looked closely at that rule (I suppose he does what it's > intended to do ;-) ), but why not call automake once in $srcdir then > once in $top_srcdir? > TBH I'd rather not spend ages figuring out all this magic :-) > > But yeah, maybe some careful looking at these rules might lead to a > couple of simple shell lines. > I looked a bit more closely at gcc, and noticed that ACLOCAL_AMFLAGS is given different values at various parts of the source tree: -I $(top_srcdir) -I $(top_srcdir)/config -I ../config -I ../config -I .. -I ./config -I ../config -I .. -I ../../config -I .. -I ../config -I ../.. -I ../../config -I . -I .. -I ../config -I m4 not sure if the current autoregen.py is in sync with that? Also... I discovered the existence of an automake rule: am--refresh which IIUC is intended to automake the update of Makefile and its dependencies. I'm by no means an autotool expert :-) Christophe > > > > > > What is currently > > > missing is a way to easily regenerate files without having to run a > > > full 'make all' (which currently takes care of calling autoconf & > > > friends to update configure/Makefile.in). > > > > > > But yeah, having to configure before being able to regenerate files is > > > a bit awkward too :-) > > > > I understand the constraints your are working with, and I guess that > > doing: > > > > ./configure && make regenerate > > > > is not too bad. The buildbot could probably do that... except that > > it would need a way to force regenerate everything, ignoring the > > timestamps. Perhaps this option of GNU make would work? > > > > -B, --always-make > > Unconditionally make all targets. > I noticed that option when writing my previous message, maybe that would work. > > > >> Looking at the rule to re-generate copying.c in gdb for instance: > > >> > > >> # Make copying.c from COPYING > > >> $(srcdir)/copying.c: @MAINTAINER_MODE_TRUE@ $(srcdir)/../COPYING3 $(srcdir)/copying.awk > > >> awk -f $(srcdir)/copying.awk \ > > >> < $(srcdir)/../COPYING3 > $(srcdir)/copying.tmp > > >> mv $(srcdir)/copying.tmp $(srcdir)/copying.c > > >> > > >> There is nothing in this code that requires having configured the source > > >> tree. This code could for instance be moved to some > > >> generate-copying-c.sh script. generate-copying-c.sh could be called by > > >> an hypothetical autoregen.sh script, as well as the copying.c Makefile > > >> target, if we want to continue supporting the maintainer mode. > > > Wouldn't it be more obscure than now? Currently such build rules are > > > all in the relevant Makefile. You'd have to open several scripts to > > > discover what's involved with updating copying.c > > > > Maybe, that's subjective :). The logic to regenerate would be in one > > script, and yes that script could be invoked from different places. At > > least the paper trail would be relatively easy to follow. > > > > >> Much like your regenerate targets, an autoregen.sh script in a given > > >> directory would be responsible to re-generate all the files in this > > >> directory that are generated and checked in git. It would also be > > >> responsible to call any autoregen.sh file in subdirectories. > > > Makefiles already have all that in place :-) > > > Except if you consider that you'd want to ignore timestamps and always > > > regenerate things? > > > > Yeah, for the buildbot autoregen job, since its job it to verify that > > everything has been generated correctly, we would like to regenerate > > everything, regardless of the timestamps. > The bot I want to put in place would regenerate things as they are > supposed to be, then build and run the testsuite to make sure that > what is supposed to be committed would work (if the committer > regenerates everything correctly) > > > > > Speaking of "already have all that in place", maintaining a script in > > the buildbot to re-implement all the re-generation logic is the worst of > > all, so I would love to avoid having to reimplemement some of that logic > > out of the repo :). > agreed! > autoregen.py should probably be moved from the bot to the binutils-gdb > and gcc repos (but we'd have to remember to keep them in sync, like > many other files already...) > > > >> compiled. When experimenting with maintainer mode the other day, I > > >> stumbled on the opcodes/i386-gen, for instance. I don't have a good > > >> solution to that, except to rewrite these tools in a scripting language > > >> like Python. > > > > > > So for opcodes, it currently means rewriting such programs for i386, > > > aarch64, ia64 and luckily msp430/rl78/rx share the same opc2c > > > generator. > > > Not sure how to find volunteers? > > > > I would love to do that if I had infinite time :). Of course, the > > current state of things + finite amount of resources are some contraints > > you are working with, and I completely undertand that. > > :-) > > Thanks, > > Christophe > > > > > Simon