From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca (simark.ca [158.69.221.121]) by sourceware.org (Postfix) with ESMTPS id C03F53858D38 for ; Sat, 16 Mar 2024 17:16:17 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C03F53858D38 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=simark.ca Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=simark.ca ARC-Filter: OpenARC Filter v1.0.0 sourceware.org C03F53858D38 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=158.69.221.121 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710609380; cv=none; b=vTcp775OlALDELGjKrxh0xyjiEEqH4/uoETegVlcNrwcSrJ+TRthorHh3rabjju2+Wj1gFWh/duPfHYSwF95yDFYYQN4spjKdPrReClhekqWxC3lot/m463o7wy3luZFF81Ft+p7NmaMSeaHNpLfrS6LOIEtAV4X5whlk6mA2to= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710609380; c=relaxed/simple; bh=ZY4fIAzGQSkmVdbYOLcj/2B2cbFUbuT9Bkq7X7L8+Fs=; h=DKIM-Signature:Message-ID:Date:MIME-Version:Subject:To:From; b=NFW0ZAZaUjkxlmfNHQk1o0X338cPKRmJ99oNbtz+AM+5OPnKMTnRTUH2PcHagMWHCwM9q1K59o9pyX/chj+nuENqcEw082R4caeLMyYLnYeL5qQi4oTGxroQqyxasfxYvacJBSV+cyRu6fyW0cQK/12b818wMi/7KmauUJPv1Mg= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=simark.ca; s=mail; t=1710609377; bh=ZY4fIAzGQSkmVdbYOLcj/2B2cbFUbuT9Bkq7X7L8+Fs=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=IQF3siSksx0oJfWYsNjWICnh2cL+a84/bKpqvGR7ci4BWawuZSiYMBqHApBmh1J4E Qg/bosjdsiKIXqMQJJguipKFDb4xnkGEVJRpFmvHGRMMC547QJIE1NnsP+i6J86IkD jBN18VPYZ4fsXTlWbBmkOXD1ErTZmdrgkuW+SQU4= Received: from [10.0.0.11] (modemcable238.237-201-24.mc.videotron.ca [24.201.237.238]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPSA id 744801E01D; Sat, 16 Mar 2024 13:16:16 -0400 (EDT) Message-ID: Date: Sat, 16 Mar 2024 13:16:15 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC] add regenerate Makefile target Content-Language: en-US To: Christophe Lyon Cc: binutils@sourceware.org, gdb@sourceware.org, gcc@gcc.gnu.org References: <20240313080237.1143034-1-christophe.lyon@linaro.org> <1eb529f2-3842-4090-a8e2-f713a28f2394@simark.ca> From: Simon Marchi In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 2024-03-15 04:50, Christophe Lyon via Gdb wrote: > On Thu, 14 Mar 2024 at 19:10, Simon Marchi wrote: >> My first thought it: why is it a Makefile target, instead of some script >> on the side (like autoregen.sh). It would be nice / useful to be >> able to it without configuring / building anything. For instance, the >> autoregen buildbot job could run it without configuring anything. >> Ideally, the buildbot wouldn't maintain its own autoregen.py file on the >> side, it would just use whatever is in the repo. > > Firstly because of what you mention later: some regeneration steps > require building host tools first, like the XXX-gen in opcodes. "build" and not "host", I think? > Since the existing Makefiles already contain the rules to autoregen > all these files, it seemed natural to me to reuse them, to avoid > reinventing the wheel with the risk of introducing new bugs. I understand. Although one advantage of moving the actual code out of the Makefile (even if there's still a Makefile rule calling the external script), is that it's much easier to maintain. Editors are much more useful when editing a standalone shell script than editing shell code in a Makefile target. It doesn't have to be this big one liner if you want to use variables, you don't need to escape $, you can run it through linters, you can call it by hand, etc. This is what I did here, for instance: https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=f39632d9579d3c97f1e50a728efed3c5409747d2 So I think there's value in any case of moving the regeneration logic out of the Makefiles per se. > This involves changes in places where I've never looked at before, so > I'd rather reuse as much existing support as possible. > > For instance, there are the generators in opcodes/, but also things in > sim/, bfd/, updates to the docs and potfiles. In gcc, there's also > something "unusual" in fixincludes/ and libgfortran/ > > In fact, I considered also including 'configure', 'Makefile.in', > etc... in the 'regenerate' target, it does not seem natural to me to > invoke a script on the side, where you have to replicate the behaviour > of existing Makefiles, possibly getting out-of-sync when someone > forgets to update either Makefile or autoregen.py. I'm not sure I follow. Are you referring to the rules that automake automatically puts to re-generate Makefile.in and others when Makefile.am has changed? Your regenerate target would depend on those builtin rules? Let's say my generate-autostuff.sh script does: aclocal --some-flags automake --some-other-flags autoconf --some-other-other-flags And the checked-in Makefile.in is regenerated based on that. Wouldn't the built-in rules just call aclocal/automake/autoconf with those same flags? I don't see why they would get out of sync. > What is currently > missing is a way to easily regenerate files without having to run a > full 'make all' (which currently takes care of calling autoconf & > friends to update configure/Makefile.in). > > But yeah, having to configure before being able to regenerate files is > a bit awkward too :-) I understand the constraints your are working with, and I guess that doing: ./configure && make regenerate is not too bad. The buildbot could probably do that... except that it would need a way to force regenerate everything, ignoring the timestamps. Perhaps this option of GNU make would work? -B, --always-make Unconditionally make all targets. >> Looking at the rule to re-generate copying.c in gdb for instance: >> >> # Make copying.c from COPYING >> $(srcdir)/copying.c: @MAINTAINER_MODE_TRUE@ $(srcdir)/../COPYING3 $(srcdir)/copying.awk >> awk -f $(srcdir)/copying.awk \ >> < $(srcdir)/../COPYING3 > $(srcdir)/copying.tmp >> mv $(srcdir)/copying.tmp $(srcdir)/copying.c >> >> There is nothing in this code that requires having configured the source >> tree. This code could for instance be moved to some >> generate-copying-c.sh script. generate-copying-c.sh could be called by >> an hypothetical autoregen.sh script, as well as the copying.c Makefile >> target, if we want to continue supporting the maintainer mode. > Wouldn't it be more obscure than now? Currently such build rules are > all in the relevant Makefile. You'd have to open several scripts to > discover what's involved with updating copying.c Maybe, that's subjective :). The logic to regenerate would be in one script, and yes that script could be invoked from different places. At least the paper trail would be relatively easy to follow. >> Much like your regenerate targets, an autoregen.sh script in a given >> directory would be responsible to re-generate all the files in this >> directory that are generated and checked in git. It would also be >> responsible to call any autoregen.sh file in subdirectories. > Makefiles already have all that in place :-) > Except if you consider that you'd want to ignore timestamps and always > regenerate things? Yeah, for the buildbot autoregen job, since its job it to verify that everything has been generated correctly, we would like to regenerate everything, regardless of the timestamps. Speaking of "already have all that in place", maintaining a script in the buildbot to re-implement all the re-generation logic is the worst of all, so I would love to avoid having to reimplemement some of that logic out of the repo :). >> compiled. When experimenting with maintainer mode the other day, I >> stumbled on the opcodes/i386-gen, for instance. I don't have a good >> solution to that, except to rewrite these tools in a scripting language >> like Python. > > So for opcodes, it currently means rewriting such programs for i386, > aarch64, ia64 and luckily msp430/rl78/rx share the same opc2c > generator. > Not sure how to find volunteers? I would love to do that if I had infinite time :). Of course, the current state of things + finite amount of resources are some contraints you are working with, and I completely undertand that. Simon