Re: [RFC] add regenerate Makefile target

[Simon Marchi <simark@simark.ca>]

On 2024-03-15 04:50, Christophe Lyon via Gdb wrote:
> On Thu, 14 Mar 2024 at 19:10, Simon Marchi <simark@simark.ca> wrote:
>> My first thought it: why is it a Makefile target, instead of some script
>> on the side (like autoregen.sh).  It would be nice / useful to be
>> able to it without configuring / building anything.  For instance, the
>> autoregen buildbot job could run it without configuring anything.
>> Ideally, the buildbot wouldn't maintain its own autoregen.py file on the
>> side, it would just use whatever is in the repo.
> 
> Firstly because of what you mention later: some regeneration steps
> require building host tools first, like the XXX-gen in opcodes.

"build" and not "host", I think?

> Since the existing Makefiles already contain the rules to autoregen
> all these files, it seemed natural to me to reuse them, to avoid
> reinventing the wheel with the risk of introducing new bugs.

I understand.  Although one advantage of moving the actual code out of
the Makefile (even if there's still a Makefile rule calling the external
script), is that it's much easier to maintain.  Editors are much more
useful when editing a standalone shell script than editing shell code in
a Makefile target.  It doesn't have to be this big one liner if you want
to use variables, you don't need to escape $, you can run it through
linters, you can call it by hand, etc.  This is what I did here, for
instance:

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=f39632d9579d3c97f1e50a728efed3c5409747d2

So I think there's value in any case of moving the regeneration logic
out of the Makefiles per se.

> This involves changes in places where I've never looked at before, so
> I'd rather reuse as much existing support as possible.
> 
> For instance, there are the generators in opcodes/, but also things in
> sim/, bfd/, updates to the docs and potfiles. In gcc, there's also
> something "unusual" in fixincludes/ and libgfortran/
> 
> In fact, I considered also including 'configure', 'Makefile.in',
> etc... in the 'regenerate' target, it does not seem natural to me to
> invoke a script on the side, where you have to replicate the behaviour
> of existing Makefiles, possibly getting out-of-sync when someone
> forgets to update either Makefile or autoregen.py.

I'm not sure I follow.  Are you referring to the rules that automake
automatically puts to re-generate Makefile.in and others when
Makefile.am has changed?  Your regenerate target would depend on those
builtin rules?

Let's say my generate-autostuff.sh script does:

  aclocal --some-flags
  automake --some-other-flags
  autoconf --some-other-other-flags

And the checked-in Makefile.in is regenerated based on that.  Wouldn't
the built-in rules just call aclocal/automake/autoconf with those same
flags?  I don't see why they would get out of sync.

> What is currently
> missing is a way to easily regenerate files without having to run a
> full 'make all' (which currently takes care of calling autoconf &
> friends to update configure/Makefile.in).
> 
> But yeah, having to configure before being able to regenerate files is
> a bit awkward too :-)

I understand the constraints your are working with, and I guess that
doing:

  ./configure && make regenerate

is not too bad.  The buildbot could probably do that... except that
it would need a way to force regenerate everything, ignoring the
timestamps.  Perhaps this option of GNU make would work?

       -B, --always-make
            Unconditionally make all targets.

>> Looking at the rule to re-generate copying.c in gdb for instance:
>>
>>     # Make copying.c from COPYING
>>     $(srcdir)/copying.c: @MAINTAINER_MODE_TRUE@ $(srcdir)/../COPYING3 $(srcdir)/copying.awk
>>            awk -f $(srcdir)/copying.awk \
>>                < $(srcdir)/../COPYING3 > $(srcdir)/copying.tmp
>>            mv $(srcdir)/copying.tmp $(srcdir)/copying.c
>>
>> There is nothing in this code that requires having configured the source
>> tree.  This code could for instance be moved to some
>> generate-copying-c.sh script.  generate-copying-c.sh could be called by
>> an hypothetical autoregen.sh script, as well as the copying.c Makefile
>> target, if we want to continue supporting the maintainer mode.
> Wouldn't it be more obscure than now? Currently such build rules are
> all in the relevant Makefile. You'd have to open several scripts to
> discover what's involved with updating copying.c

Maybe, that's subjective :).  The logic to regenerate would be in one
script, and yes that script could be invoked from different places.  At
least the paper trail would be relatively easy to follow.

>> Much like your regenerate targets, an autoregen.sh script in a given
>> directory would be responsible to re-generate all the files in this
>> directory that are generated and checked in git.  It would also be
>> responsible to call any autoregen.sh file in subdirectories.
> Makefiles already have all that in place :-)
> Except if you consider that you'd want to ignore timestamps and always
> regenerate things?

Yeah, for the buildbot autoregen job, since its job it to verify that
everything has been generated correctly, we would like to regenerate
everything, regardless of the timestamps.

Speaking of "already have all that in place", maintaining a script in
the buildbot to re-implement all the re-generation logic is the worst of
all, so I would love to avoid having to reimplemement some of that logic
out of the repo :).

>> compiled.  When experimenting with maintainer mode the other day, I
>> stumbled on the opcodes/i386-gen, for instance.  I don't have a good
>> solution to that, except to rewrite these tools in a scripting language
>> like Python.
> 
> So for opcodes, it currently means rewriting such programs for i386,
> aarch64, ia64 and luckily msp430/rl78/rx share the same opc2c
> generator.
> Not sure how to find volunteers?

I would love to do that if I had infinite time :).  Of course, the
current state of things + finite amount of resources are some contraints
you are working with, and I completely undertand that.

Simon