Re: [RFC] add regenerate Makefile target

[Christophe Lyon <christophe.lyon@linaro.org>]

On Sat, 16 Mar 2024 at 18:16, Simon Marchi <simark@simark.ca> wrote:
>
>
>
> On 2024-03-15 04:50, Christophe Lyon via Gdb wrote:
> > On Thu, 14 Mar 2024 at 19:10, Simon Marchi <simark@simark.ca> wrote:
> >> My first thought it: why is it a Makefile target, instead of some script
> >> on the side (like autoregen.sh).  It would be nice / useful to be
> >> able to it without configuring / building anything.  For instance, the
> >> autoregen buildbot job could run it without configuring anything.
> >> Ideally, the buildbot wouldn't maintain its own autoregen.py file on the
> >> side, it would just use whatever is in the repo.
> >
> > Firstly because of what you mention later: some regeneration steps
> > require building host tools first, like the XXX-gen in opcodes.
>
> "build" and not "host", I think?

yes, sorry

> > Since the existing Makefiles already contain the rules to autoregen
> > all these files, it seemed natural to me to reuse them, to avoid
> > reinventing the wheel with the risk of introducing new bugs.
>
> I understand.  Although one advantage of moving the actual code out of
> the Makefile (even if there's still a Makefile rule calling the external
> script), is that it's much easier to maintain.  Editors are much more
> useful when editing a standalone shell script than editing shell code in
> a Makefile target.  It doesn't have to be this big one liner if you want
> to use variables, you don't need to escape $, you can run it through
> linters, you can call it by hand, etc.  This is what I did here, for
> instance:
>
> https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=f39632d9579d3c97f1e50a728efed3c5409747d2
>
> So I think there's value in any case of moving the regeneration logic
> out of the Makefiles per se.
>
In this case, the generation rules look simple enough indeed.
But as mentioned elsewhere in the thread, there are more complex
cases, which involve building helper tools, which have dependencies on
bfd and libiberty for instance. I'm not sure that's easily/naturally
scriptable?
There's also 'chew' in bfd/

> > This involves changes in places where I've never looked at before, so
> > I'd rather reuse as much existing support as possible.
> >
> > For instance, there are the generators in opcodes/, but also things in
> > sim/, bfd/, updates to the docs and potfiles. In gcc, there's also
> > something "unusual" in fixincludes/ and libgfortran/
> >
> > In fact, I considered also including 'configure', 'Makefile.in',
> > etc... in the 'regenerate' target, it does not seem natural to me to
> > invoke a script on the side, where you have to replicate the behaviour
> > of existing Makefiles, possibly getting out-of-sync when someone
> > forgets to update either Makefile or autoregen.py.
>
> I'm not sure I follow.  Are you referring to the rules that automake
> automatically puts to re-generate Makefile.in and others when
> Makefile.am has changed?  Your regenerate target would depend on those
> builtin rules?
Yes, "regenerate" would include "configure, Makeifile.in, configh"
(as/if needed) in its list of dependencies.

>
> Let's say my generate-autostuff.sh script does:
>
>   aclocal --some-flags
>   automake --some-other-flags
>   autoconf --some-other-other-flags
>
> And the checked-in Makefile.in is regenerated based on that.  Wouldn't
> the built-in rules just call aclocal/automake/autoconf with those same
> flags?  I don't see why they would get out of sync.
Well the rule to regenerate Makefile.in (eg in in opcodes/) is a bit
more complex
than just calling automake. IIUC it calls automake --foreign it any of
*.m4 file from $(am__configure_deps) that is newer than Makefile.in
(with an early exit in the loop), does nothing if Makefile.am or
doc/local.mk are newer than Makefile.in, and then calls 'automake
--foreign Makefile'

I've never looked closely at that rule (I suppose he does what it's
intended to do ;-) ), but why not call automake once in $srcdir then
once in $top_srcdir?
TBH I'd rather not spend ages figuring out all this magic :-)

But yeah, maybe some careful looking at these rules might lead to a
couple of simple shell lines.


>
> > What is currently
> > missing is a way to easily regenerate files without having to run a
> > full 'make all' (which currently takes care of calling autoconf &
> > friends to update configure/Makefile.in).
> >
> > But yeah, having to configure before being able to regenerate files is
> > a bit awkward too :-)
>
> I understand the constraints your are working with, and I guess that
> doing:
>
>   ./configure && make regenerate
>
> is not too bad.  The buildbot could probably do that... except that
> it would need a way to force regenerate everything, ignoring the
> timestamps.  Perhaps this option of GNU make would work?
>
>        -B, --always-make
>             Unconditionally make all targets.
I noticed that option when writing my previous message, maybe that would work.

> >> Looking at the rule to re-generate copying.c in gdb for instance:
> >>
> >>     # Make copying.c from COPYING
> >>     $(srcdir)/copying.c: @MAINTAINER_MODE_TRUE@ $(srcdir)/../COPYING3 $(srcdir)/copying.awk
> >>            awk -f $(srcdir)/copying.awk \
> >>                < $(srcdir)/../COPYING3 > $(srcdir)/copying.tmp
> >>            mv $(srcdir)/copying.tmp $(srcdir)/copying.c
> >>
> >> There is nothing in this code that requires having configured the source
> >> tree.  This code could for instance be moved to some
> >> generate-copying-c.sh script.  generate-copying-c.sh could be called by
> >> an hypothetical autoregen.sh script, as well as the copying.c Makefile
> >> target, if we want to continue supporting the maintainer mode.
> > Wouldn't it be more obscure than now? Currently such build rules are
> > all in the relevant Makefile. You'd have to open several scripts to
> > discover what's involved with updating copying.c
>
> Maybe, that's subjective :).  The logic to regenerate would be in one
> script, and yes that script could be invoked from different places.  At
> least the paper trail would be relatively easy to follow.
>
> >> Much like your regenerate targets, an autoregen.sh script in a given
> >> directory would be responsible to re-generate all the files in this
> >> directory that are generated and checked in git.  It would also be
> >> responsible to call any autoregen.sh file in subdirectories.
> > Makefiles already have all that in place :-)
> > Except if you consider that you'd want to ignore timestamps and always
> > regenerate things?
>
> Yeah, for the buildbot autoregen job, since its job it to verify that
> everything has been generated correctly, we would like to regenerate
> everything, regardless of the timestamps.
The bot I want to put in place would regenerate things as they are
supposed to be, then build and run the testsuite to make sure that
what is supposed to be committed would work (if the committer
regenerates everything correctly)



> Speaking of "already have all that in place", maintaining a script in
> the buildbot to re-implement all the re-generation logic is the worst of
> all, so I would love to avoid having to reimplemement some of that logic
> out of the repo :).
agreed!
autoregen.py should probably be moved from the bot to the binutils-gdb
and gcc repos (but we'd have to remember to keep them in sync, like
many other files already...)

> >> compiled.  When experimenting with maintainer mode the other day, I
> >> stumbled on the opcodes/i386-gen, for instance.  I don't have a good
> >> solution to that, except to rewrite these tools in a scripting language
> >> like Python.
> >
> > So for opcodes, it currently means rewriting such programs for i386,
> > aarch64, ia64 and luckily msp430/rl78/rx share the same opc2c
> > generator.
> > Not sure how to find volunteers?
>
> I would love to do that if I had infinite time :).  Of course, the
> current state of things + finite amount of resources are some contraints
> you are working with, and I completely undertand that.

:-)

Thanks,

Christophe

>
> Simon