Re: [RFC] add regenerate Makefile target

[Christophe Lyon <christophe.lyon@linaro.org>]

On Fri, 15 Mar 2024 at 15:13, Eric Gallager <egall@gwmail.gwu.edu> wrote:
>
> On Fri, Mar 15, 2024 at 4:53 AM Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
> >
> > On Thu, 14 Mar 2024 at 19:10, Simon Marchi <simark@simark.ca> wrote:
> > >
> > >
> > >
> > > On 2024-03-13 04:02, Christophe Lyon via Gdb wrote:
> > > > Hi!
> > > >
> > > > After recent discussions on IRC and on the lists about maintainer-mode
> > > > and various problems with auto-generated source files, I've written
> > > > this small prototype.
> > > >
> > > > Based on those discussions, I assumed that people generally want to
> > > > update autotools files using a script similar to autoregen.py, which
> > > > takes care of running aclocal, autoheader, automake and autoconf as
> > > > appropriate.
> > > >
> > > > What is currently missing is a "simple" way of regenerating other
> > > > files, which happens normally with --enable-maintainer-mode (which is
> > > > reportedly broken).  This patch as a "regenerate" Makefile target
> > > > which can be called to update those files, provided
> > > > --enable-maintainer-mode is used.
> > > >
> > > > I tried this approach with the following workflow for binutils/gdb:
> > > > - run autoregen.py in srcdir
> > > > - cd builddir
> > > > - configure --enable-maintainer-mode
> > > > - make all-bfd all-libiberty regenerate -j1
> > > > - for gdb: make all -C gdb/data-directory -j1
> > > > - make all -jXXX
> > > >
> > > > Making 'all' in bfd and libiberty is needed by some XXX-gen host
> > > > programs in opcodes.
> > > >
> > > > The advantage (for instance for CI) is that we can regenerate files at
> > > > -j1, thus avoiding the existing race conditions, and build the rest
> > > > with -j XXX.
> > > >
> > > > Among drawbacks:
> > > > - most sub-components use Makefile.am, but gdb does not: this may make
> > > >   maintenance more complex (different rules for different projects)
> > > > - maintaining such ad-hoc "regenerate" rules would require special
> > > >   attention from maintainers/reviewers
> > > > - dependency on -all-bfd and all-libiberty is probably not fully
> > > >    intuitive, but should not be a problem if the "regenerate" rules
> > > >    are used after a full build for instance
> > > >
> > > > Of course Makefile.def/Makefile.tpl would need further cleanup as I
> > > > didn't try to take gcc into account is this patch.
> > > >
> > > > Thoughts?
> > >
> > > My first thought it: why is it a Makefile target, instead of some script
> > > on the side (like autoregen.sh).  It would be nice / useful to be
> > > able to it without configuring / building anything.  For instance, the
> > > autoregen buildbot job could run it without configuring anything.
> > > Ideally, the buildbot wouldn't maintain its own autoregen.py file on the
> > > side, it would just use whatever is in the repo.
> >
> > Firstly because of what you mention later: some regeneration steps
> > require building host tools first, like the XXX-gen in opcodes.
> >
> > Since the existing Makefiles already contain the rules to autoregen
> > all these files, it seemed natural to me to reuse them, to avoid
> > reinventing the wheel with the risk of introducing new bugs.
> >
> > This involves changes in places where I've never looked at before, so
> > I'd rather reuse as much existing support as possible.
> >
> > For instance, there are the generators in opcodes/, but also things in
> > sim/, bfd/, updates to the docs and potfiles. In gcc, there's also
> > something "unusual" in fixincludes/ and libgfortran/
> >
> > In fact, I considered also including 'configure', 'Makefile.in',
> > etc... in the 'regenerate' target, it does not seem natural to me to
> > invoke a script on the side, where you have to replicate the behaviour
> > of existing Makefiles, possibly getting out-of-sync when someone
> > forgets to update either Makefile or autoregen.py. What is currently
> > missing is a way to easily regenerate files without having to run a
> > full 'make all' (which currently takes care of calling autoconf &
> > friends to update configure/Makefile.in).
> >
> > But yeah, having to configure before being able to regenerate files is
> > a bit awkward too :-)
> >
> >
> > >
> > > Looking at the rule to re-generate copying.c in gdb for instance:
> > >
> > >     # Make copying.c from COPYING
> > >     $(srcdir)/copying.c: @MAINTAINER_MODE_TRUE@ $(srcdir)/../COPYING3 $(srcdir)/copying.awk
> > >            awk -f $(srcdir)/copying.awk \
> > >                < $(srcdir)/../COPYING3 > $(srcdir)/copying.tmp
> > >            mv $(srcdir)/copying.tmp $(srcdir)/copying.c
> > >
> > > There is nothing in this code that requires having configured the source
> > > tree.  This code could for instance be moved to some
> > > generate-copying-c.sh script.  generate-copying-c.sh could be called by
> > > an hypothetical autoregen.sh script, as well as the copying.c Makefile
> > > target, if we want to continue supporting the maintainer mode.
> > Wouldn't it be more obscure than now? Currently such build rules are
> > all in the relevant Makefile. You'd have to open several scripts to
> > discover what's involved with updating copying.c
> >
>
> Yeah I agree that it's good to keep all build rules in the Makefile;
> if there's a possibility of something changing, things that depend
> upon it need to know, and the best way to express those dependencies
> is in the Makefile.
>
> > >
> > > Much like your regenerate targets, an autoregen.sh script in a given
> > > directory would be responsible to re-generate all the files in this
> > > directory that are generated and checked in git.  It would also be
> > > responsible to call any autoregen.sh file in subdirectories.
> > Makefiles already have all that in place :-)
> > Except if you consider that you'd want to ignore timestamps and always
> > regenerate things?
> >
> >
> > > There's just the issue of files that are generated using tools that are
> > > compiled.  When experimenting with maintainer mode the other day, I
> > > stumbled on the opcodes/i386-gen, for instance.  I don't have a good
> > > solution to that, except to rewrite these tools in a scripting language
> > > like Python.
> >
> > So for opcodes, it currently means rewriting such programs for i386,
> > aarch64, ia64 and luckily msp430/rl78/rx share the same opc2c
> > generator.
>
> Also there are the files generated by cgen, too, which no one seems to

I had heard about cgen ages ago but I had forgotten about them.
And your message made me discover configure's option --enable-cgen-maint :-)

> know how to regenerate, either. And then in bfd there's that chew
> program in the doc subdir. And then in the binutils subdirectory
> proper there's that sysinfo tool for generating sysroff.[ch].
>
> > Not sure how to find volunteers?
> >
> > Christophe
> >
> > >
> > > Simon