Re: [RFC] add regenerate Makefile target

[Christophe Lyon <christophe.lyon@linaro.org>]

Hi,

On Mon, 18 Mar 2024 at 18:25, Christophe Lyon
<christophe.lyon@linaro.org> wrote:
>
> On Sat, 16 Mar 2024 at 18:16, Simon Marchi <simark@simark.ca> wrote:
> >
> >
> >
> > On 2024-03-15 04:50, Christophe Lyon via Gdb wrote:
> > > On Thu, 14 Mar 2024 at 19:10, Simon Marchi <simark@simark.ca> wrote:
> > >> My first thought it: why is it a Makefile target, instead of some script
> > >> on the side (like autoregen.sh).  It would be nice / useful to be
> > >> able to it without configuring / building anything.  For instance, the
> > >> autoregen buildbot job could run it without configuring anything.
> > >> Ideally, the buildbot wouldn't maintain its own autoregen.py file on the
> > >> side, it would just use whatever is in the repo.
> > >
> > > Firstly because of what you mention later: some regeneration steps
> > > require building host tools first, like the XXX-gen in opcodes.
> >
> > "build" and not "host", I think?
>
> yes, sorry
>
> > > Since the existing Makefiles already contain the rules to autoregen
> > > all these files, it seemed natural to me to reuse them, to avoid
> > > reinventing the wheel with the risk of introducing new bugs.
> >
> > I understand.  Although one advantage of moving the actual code out of
> > the Makefile (even if there's still a Makefile rule calling the external
> > script), is that it's much easier to maintain.  Editors are much more
> > useful when editing a standalone shell script than editing shell code in
> > a Makefile target.  It doesn't have to be this big one liner if you want
> > to use variables, you don't need to escape $, you can run it through
> > linters, you can call it by hand, etc.  This is what I did here, for
> > instance:
> >
> > https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=f39632d9579d3c97f1e50a728efed3c5409747d2
> >
> > So I think there's value in any case of moving the regeneration logic
> > out of the Makefiles per se.
> >
> In this case, the generation rules look simple enough indeed.
> But as mentioned elsewhere in the thread, there are more complex
> cases, which involve building helper tools, which have dependencies on
> bfd and libiberty for instance. I'm not sure that's easily/naturally
> scriptable?
> There's also 'chew' in bfd/
>
> > > This involves changes in places where I've never looked at before, so
> > > I'd rather reuse as much existing support as possible.
> > >
> > > For instance, there are the generators in opcodes/, but also things in
> > > sim/, bfd/, updates to the docs and potfiles. In gcc, there's also
> > > something "unusual" in fixincludes/ and libgfortran/
> > >
> > > In fact, I considered also including 'configure', 'Makefile.in',
> > > etc... in the 'regenerate' target, it does not seem natural to me to
> > > invoke a script on the side, where you have to replicate the behaviour
> > > of existing Makefiles, possibly getting out-of-sync when someone
> > > forgets to update either Makefile or autoregen.py.
> >
> > I'm not sure I follow.  Are you referring to the rules that automake
> > automatically puts to re-generate Makefile.in and others when
> > Makefile.am has changed?  Your regenerate target would depend on those
> > builtin rules?
> Yes, "regenerate" would include "configure, Makeifile.in, configh"
> (as/if needed) in its list of dependencies.
>
> >
> > Let's say my generate-autostuff.sh script does:
> >
> >   aclocal --some-flags
> >   automake --some-other-flags
> >   autoconf --some-other-other-flags
> >
> > And the checked-in Makefile.in is regenerated based on that.  Wouldn't
> > the built-in rules just call aclocal/automake/autoconf with those same
> > flags?  I don't see why they would get out of sync.
> Well the rule to regenerate Makefile.in (eg in in opcodes/) is a bit
> more complex
> than just calling automake. IIUC it calls automake --foreign it any of
> *.m4 file from $(am__configure_deps) that is newer than Makefile.in
> (with an early exit in the loop), does nothing if Makefile.am or
> doc/local.mk are newer than Makefile.in, and then calls 'automake
> --foreign Makefile'
>
> I've never looked closely at that rule (I suppose he does what it's
> intended to do ;-) ), but why not call automake once in $srcdir then
> once in $top_srcdir?
> TBH I'd rather not spend ages figuring out all this magic :-)
>
> But yeah, maybe some careful looking at these rules might lead to a
> couple of simple shell lines.
>

I looked a bit more closely at gcc, and noticed that ACLOCAL_AMFLAGS
is given different values at various parts of the source tree:
-I $(top_srcdir) -I $(top_srcdir)/config
-I ../config
-I ../config -I ..
-I ./config -I ../config
-I .. -I ../../config
-I .. -I ../config
-I ../.. -I ../../config
-I . -I .. -I ../config
-I m4

not sure if the current autoregen.py is in sync with that?

Also... I discovered the existence of an automake rule:
am--refresh which IIUC is intended to automake the update of Makefile
and its dependencies.

I'm by no means an autotool expert :-)

Christophe

>
> >
> > > What is currently
> > > missing is a way to easily regenerate files without having to run a
> > > full 'make all' (which currently takes care of calling autoconf &
> > > friends to update configure/Makefile.in).
> > >
> > > But yeah, having to configure before being able to regenerate files is
> > > a bit awkward too :-)
> >
> > I understand the constraints your are working with, and I guess that
> > doing:
> >
> >   ./configure && make regenerate
> >
> > is not too bad.  The buildbot could probably do that... except that
> > it would need a way to force regenerate everything, ignoring the
> > timestamps.  Perhaps this option of GNU make would work?
> >
> >        -B, --always-make
> >             Unconditionally make all targets.
> I noticed that option when writing my previous message, maybe that would work.
>
> > >> Looking at the rule to re-generate copying.c in gdb for instance:
> > >>
> > >>     # Make copying.c from COPYING
> > >>     $(srcdir)/copying.c: @MAINTAINER_MODE_TRUE@ $(srcdir)/../COPYING3 $(srcdir)/copying.awk
> > >>            awk -f $(srcdir)/copying.awk \
> > >>                < $(srcdir)/../COPYING3 > $(srcdir)/copying.tmp
> > >>            mv $(srcdir)/copying.tmp $(srcdir)/copying.c
> > >>
> > >> There is nothing in this code that requires having configured the source
> > >> tree.  This code could for instance be moved to some
> > >> generate-copying-c.sh script.  generate-copying-c.sh could be called by
> > >> an hypothetical autoregen.sh script, as well as the copying.c Makefile
> > >> target, if we want to continue supporting the maintainer mode.
> > > Wouldn't it be more obscure than now? Currently such build rules are
> > > all in the relevant Makefile. You'd have to open several scripts to
> > > discover what's involved with updating copying.c
> >
> > Maybe, that's subjective :).  The logic to regenerate would be in one
> > script, and yes that script could be invoked from different places.  At
> > least the paper trail would be relatively easy to follow.
> >
> > >> Much like your regenerate targets, an autoregen.sh script in a given
> > >> directory would be responsible to re-generate all the files in this
> > >> directory that are generated and checked in git.  It would also be
> > >> responsible to call any autoregen.sh file in subdirectories.
> > > Makefiles already have all that in place :-)
> > > Except if you consider that you'd want to ignore timestamps and always
> > > regenerate things?
> >
> > Yeah, for the buildbot autoregen job, since its job it to verify that
> > everything has been generated correctly, we would like to regenerate
> > everything, regardless of the timestamps.
> The bot I want to put in place would regenerate things as they are
> supposed to be, then build and run the testsuite to make sure that
> what is supposed to be committed would work (if the committer
> regenerates everything correctly)
>
>
>
> > Speaking of "already have all that in place", maintaining a script in
> > the buildbot to re-implement all the re-generation logic is the worst of
> > all, so I would love to avoid having to reimplemement some of that logic
> > out of the repo :).
> agreed!
> autoregen.py should probably be moved from the bot to the binutils-gdb
> and gcc repos (but we'd have to remember to keep them in sync, like
> many other files already...)
>
> > >> compiled.  When experimenting with maintainer mode the other day, I
> > >> stumbled on the opcodes/i386-gen, for instance.  I don't have a good
> > >> solution to that, except to rewrite these tools in a scripting language
> > >> like Python.
> > >
> > > So for opcodes, it currently means rewriting such programs for i386,
> > > aarch64, ia64 and luckily msp430/rl78/rx share the same opc2c
> > > generator.
> > > Not sure how to find volunteers?
> >
> > I would love to do that if I had infinite time :).  Of course, the
> > current state of things + finite amount of resources are some contraints
> > you are working with, and I completely undertand that.
>
> :-)
>
> Thanks,
>
> Christophe
>
> >
> > Simon