From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) by sourceware.org (Postfix) with ESMTPS id 267CB3858D37 for ; Mon, 18 Mar 2024 17:26:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 267CB3858D37 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 267CB3858D37 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::52b ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710782767; cv=none; b=XVOC7Lyj96dZFZq+J8BXXuIffY7w29vhTATCia4sxpV9tSf6xV1Hk1852cB1zFRudRZQCnN5Mm+i7R3MMDjGIYvQ3PnMpibGXDrPlLch5GtbQzeqQucOpb6oXAmpJyXECnyhDoLN2EnoFPetL2PiuMwi9ItX5YIjCwmCOGcVti4= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710782767; c=relaxed/simple; bh=CzyP6CTjhg8BGrruvYyjDJkiyAyIZ6F7QZXRoXu7NKs=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=YfEO43cikcXELSwTIIzlyxIP3Ar/Kb39PlZnYq0vmt7eGpG1pb5hijr+wA2oaXx8rfpkfQHP/ArkfAuvE7aBiUPYHKcdVKabSEkAxcE638KTN5eQVbtRlfl8ENXElFrs7FCU4bB7GWotE1rJEa7ILF5/lWXJLDfiXPXpJUnD1kE= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-568d323e7fbso1949654a12.3 for ; Mon, 18 Mar 2024 10:26:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1710782762; x=1711387562; darn=sourceware.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ZBv6fJwu0UNmC75m3BjbqI2mjOobZlZqUzhQ0by75dk=; b=TOXurNVynVKMjHgNKhRtIM/5YednxL4pgwXGWf39iGfp1Y9a8rQ3J1YOJpe4PAK3jV qb7nE+Tc2PHGbXUjbx4VHZhQQ0J4bsEIg2QwMxAcWYrOvDp43X167ofYkoi2D3wF6j3a ZUWhur3U8LqNR/ibWx+Rfgs0yS2w5194aAoNwTfX0bIv/RwHrx6KspP5jhqqYJL+1P1T kUv8L4L6WLbk1ePBhXTgec5WVy1qenyUlRlK9m+YQvlPmskwUWP42wgkodi+Bwb7Y4GK 9sVgM8QDLa0Iwb3K/xM8nfBIarq9IEkNRhIz93ltDdBx4eegRVVcZl9ZIrAJbGsuT4w4 SHmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710782762; x=1711387562; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZBv6fJwu0UNmC75m3BjbqI2mjOobZlZqUzhQ0by75dk=; b=Rd89gI0PaJjtXrCU+w7ht5FkoO39N2KbMXgbnbhYjJ0D+arDDJYW+S8tvFzzy5i47I s+KRFimCWWjJgQrdMB8SFwZX39ozSIvsp4m+cgR33tAMLr2O/5rx/U5CgvHS26YOZPsW 1Q+ozIJg5L07jX3cCOdnlPhdTqx6eH3O5uqm/mtp92r3sSGOjebB0k3KN8vTApzvIx4w 6HAREa8wa0ZXhNQxzrU3fSQp/YdjRoSrqYpNNHTNO1eM0AubML/vvEilvFLuMvtruu3d xElSXnXwrke9vlKBfgnB+J4JdmGhEcY3Y++hcML0sIJjEexV0jt/ebjOUFbrEiGR1tND knSA== X-Forwarded-Encrypted: i=1; AJvYcCW+UUINd8N0rpId1EKsgudlkqd84mQNU4JWGBqO/i+LOLcduvY0P9mAbQOn99O8dJL88ogmA6JzVB8v2UiYr8yMY1o= X-Gm-Message-State: AOJu0YypYl25W0DLBcbPwUs7ja0r11xxHaR6ddyvQJmUalFOpvbzpbxN SSpprivfrImWhv76lC0zswPPQJkw/NeDp3atUSWTYGAGKD5oiC6JcLWifuDS+USQW4qdboswwKx L2ezIbGOC5WthAwvDZApJWq/4S03T90nRiAZulw== X-Google-Smtp-Source: AGHT+IGmv8CwkyleGo7INBZKpux2/jTJE/ijHO0JtKxFrGcLqotv38fHsMAyaQ1L1D8nXggWV1OuM2TzaqM8XQrmAHg= X-Received: by 2002:a05:6402:2b8a:b0:568:9996:7cb6 with SMTP id fj10-20020a0564022b8a00b0056899967cb6mr8998699edb.14.1710782761796; Mon, 18 Mar 2024 10:26:01 -0700 (PDT) MIME-Version: 1.0 References: <20240313080237.1143034-1-christophe.lyon@linaro.org> <1eb529f2-3842-4090-a8e2-f713a28f2394@simark.ca> In-Reply-To: From: Christophe Lyon Date: Mon, 18 Mar 2024 18:25:51 +0100 Message-ID: Subject: Re: [RFC] add regenerate Makefile target To: Simon Marchi Cc: binutils@sourceware.org, gdb@sourceware.org, gcc@gcc.gnu.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Sat, 16 Mar 2024 at 18:16, Simon Marchi wrote: > > > > On 2024-03-15 04:50, Christophe Lyon via Gdb wrote: > > On Thu, 14 Mar 2024 at 19:10, Simon Marchi wrote: > >> My first thought it: why is it a Makefile target, instead of some script > >> on the side (like autoregen.sh). It would be nice / useful to be > >> able to it without configuring / building anything. For instance, the > >> autoregen buildbot job could run it without configuring anything. > >> Ideally, the buildbot wouldn't maintain its own autoregen.py file on the > >> side, it would just use whatever is in the repo. > > > > Firstly because of what you mention later: some regeneration steps > > require building host tools first, like the XXX-gen in opcodes. > > "build" and not "host", I think? yes, sorry > > Since the existing Makefiles already contain the rules to autoregen > > all these files, it seemed natural to me to reuse them, to avoid > > reinventing the wheel with the risk of introducing new bugs. > > I understand. Although one advantage of moving the actual code out of > the Makefile (even if there's still a Makefile rule calling the external > script), is that it's much easier to maintain. Editors are much more > useful when editing a standalone shell script than editing shell code in > a Makefile target. It doesn't have to be this big one liner if you want > to use variables, you don't need to escape $, you can run it through > linters, you can call it by hand, etc. This is what I did here, for > instance: > > https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=f39632d9579d3c97f1e50a728efed3c5409747d2 > > So I think there's value in any case of moving the regeneration logic > out of the Makefiles per se. > In this case, the generation rules look simple enough indeed. But as mentioned elsewhere in the thread, there are more complex cases, which involve building helper tools, which have dependencies on bfd and libiberty for instance. I'm not sure that's easily/naturally scriptable? There's also 'chew' in bfd/ > > This involves changes in places where I've never looked at before, so > > I'd rather reuse as much existing support as possible. > > > > For instance, there are the generators in opcodes/, but also things in > > sim/, bfd/, updates to the docs and potfiles. In gcc, there's also > > something "unusual" in fixincludes/ and libgfortran/ > > > > In fact, I considered also including 'configure', 'Makefile.in', > > etc... in the 'regenerate' target, it does not seem natural to me to > > invoke a script on the side, where you have to replicate the behaviour > > of existing Makefiles, possibly getting out-of-sync when someone > > forgets to update either Makefile or autoregen.py. > > I'm not sure I follow. Are you referring to the rules that automake > automatically puts to re-generate Makefile.in and others when > Makefile.am has changed? Your regenerate target would depend on those > builtin rules? Yes, "regenerate" would include "configure, Makeifile.in, configh" (as/if needed) in its list of dependencies. > > Let's say my generate-autostuff.sh script does: > > aclocal --some-flags > automake --some-other-flags > autoconf --some-other-other-flags > > And the checked-in Makefile.in is regenerated based on that. Wouldn't > the built-in rules just call aclocal/automake/autoconf with those same > flags? I don't see why they would get out of sync. Well the rule to regenerate Makefile.in (eg in in opcodes/) is a bit more complex than just calling automake. IIUC it calls automake --foreign it any of *.m4 file from $(am__configure_deps) that is newer than Makefile.in (with an early exit in the loop), does nothing if Makefile.am or doc/local.mk are newer than Makefile.in, and then calls 'automake --foreign Makefile' I've never looked closely at that rule (I suppose he does what it's intended to do ;-) ), but why not call automake once in $srcdir then once in $top_srcdir? TBH I'd rather not spend ages figuring out all this magic :-) But yeah, maybe some careful looking at these rules might lead to a couple of simple shell lines. > > > What is currently > > missing is a way to easily regenerate files without having to run a > > full 'make all' (which currently takes care of calling autoconf & > > friends to update configure/Makefile.in). > > > > But yeah, having to configure before being able to regenerate files is > > a bit awkward too :-) > > I understand the constraints your are working with, and I guess that > doing: > > ./configure && make regenerate > > is not too bad. The buildbot could probably do that... except that > it would need a way to force regenerate everything, ignoring the > timestamps. Perhaps this option of GNU make would work? > > -B, --always-make > Unconditionally make all targets. I noticed that option when writing my previous message, maybe that would work. > >> Looking at the rule to re-generate copying.c in gdb for instance: > >> > >> # Make copying.c from COPYING > >> $(srcdir)/copying.c: @MAINTAINER_MODE_TRUE@ $(srcdir)/../COPYING3 $(srcdir)/copying.awk > >> awk -f $(srcdir)/copying.awk \ > >> < $(srcdir)/../COPYING3 > $(srcdir)/copying.tmp > >> mv $(srcdir)/copying.tmp $(srcdir)/copying.c > >> > >> There is nothing in this code that requires having configured the source > >> tree. This code could for instance be moved to some > >> generate-copying-c.sh script. generate-copying-c.sh could be called by > >> an hypothetical autoregen.sh script, as well as the copying.c Makefile > >> target, if we want to continue supporting the maintainer mode. > > Wouldn't it be more obscure than now? Currently such build rules are > > all in the relevant Makefile. You'd have to open several scripts to > > discover what's involved with updating copying.c > > Maybe, that's subjective :). The logic to regenerate would be in one > script, and yes that script could be invoked from different places. At > least the paper trail would be relatively easy to follow. > > >> Much like your regenerate targets, an autoregen.sh script in a given > >> directory would be responsible to re-generate all the files in this > >> directory that are generated and checked in git. It would also be > >> responsible to call any autoregen.sh file in subdirectories. > > Makefiles already have all that in place :-) > > Except if you consider that you'd want to ignore timestamps and always > > regenerate things? > > Yeah, for the buildbot autoregen job, since its job it to verify that > everything has been generated correctly, we would like to regenerate > everything, regardless of the timestamps. The bot I want to put in place would regenerate things as they are supposed to be, then build and run the testsuite to make sure that what is supposed to be committed would work (if the committer regenerates everything correctly) > Speaking of "already have all that in place", maintaining a script in > the buildbot to re-implement all the re-generation logic is the worst of > all, so I would love to avoid having to reimplemement some of that logic > out of the repo :). agreed! autoregen.py should probably be moved from the bot to the binutils-gdb and gcc repos (but we'd have to remember to keep them in sync, like many other files already...) > >> compiled. When experimenting with maintainer mode the other day, I > >> stumbled on the opcodes/i386-gen, for instance. I don't have a good > >> solution to that, except to rewrite these tools in a scripting language > >> like Python. > > > > So for opcodes, it currently means rewriting such programs for i386, > > aarch64, ia64 and luckily msp430/rl78/rx share the same opc2c > > generator. > > Not sure how to find volunteers? > > I would love to do that if I had infinite time :). Of course, the > current state of things + finite amount of resources are some contraints > you are working with, and I completely undertand that. :-) Thanks, Christophe > > Simon