Help needed with maintainer-mode

Help needed with maintainer-mode

[Christophe Lyon]

Hi!

Sorry for cross-posting, but I'm not sure the rules/guidelines are the
same in gcc vs binutils/gdb.

TL;DR: are there some guidelines about how to use/enable maintainer-mode?

In the context of the Linaro CI, I've been looking at enabling
maintainer-mode at configure time in our configurations where we test
patches before they are committed (aka "precommit CI", which relies on
patchwork).

Indeed, auto-generated files are not part of patch submissions, and
when a patch implies regenerating some files before building, we
currently report wrong failures because we don't perform such updates.

I hoped improving this would be as simple as adding
--enable-maintainer-mode when configuring, after making sure
autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
libtool and gettext seems OK).

However, doing so triggered several problems, which look like race
conditions in the build system (we build at -j160):
- random build errors in binutils / gdb with messages like "No rule to
make target 'po/BLD-POTFILES.in". I managed to reproduce something
similar manually once, I noticed an empty Makefile; the build logs are
of course difficult to read, so I couldn't figure out yet what could
cause this.

- random build failures in gcc in fixincludes. I think this is a race
condition because fixincludes is updated concurrently both from
/fixincludes and $buillddir/fixincludes. Probably fixable in gcc
Makefiles.

- I've seen other errors when building gcc like
configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
from libquadmath. I haven't investigated this yet.

I've read binutils' README-maintainer-mode, which contains a warning
about distclean, but we don't use this: we start our builds from a
scratch directory.

So... I'm wondering if there are some "official" guidelines about how
to regenerate files, and/or use maintainer-mode?  Maybe I missed a
"magic" make target (eg 'make autoreconf-all') that should be executed
after configure and before 'make all'?

I've noticed that sourceware's buildbot has a small script
"autoregen.py" which does not use the project's build system, but
rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
Should we replicate that?

Thanks,

Christophe

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists)]

On 29/02/2024 10:22, Christophe Lyon via Gcc wrote:
> Hi!
> 
> Sorry for cross-posting, but I'm not sure the rules/guidelines are the
> same in gcc vs binutils/gdb.
> 
> TL;DR: are there some guidelines about how to use/enable maintainer-mode?
> 
> In the context of the Linaro CI, I've been looking at enabling
> maintainer-mode at configure time in our configurations where we test
> patches before they are committed (aka "precommit CI", which relies on
> patchwork).
> 
> Indeed, auto-generated files are not part of patch submissions, and
> when a patch implies regenerating some files before building, we
> currently report wrong failures because we don't perform such updates.
> 
> I hoped improving this would be as simple as adding
> --enable-maintainer-mode when configuring, after making sure
> autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
> libtool and gettext seems OK).
> 
> However, doing so triggered several problems, which look like race
> conditions in the build system (we build at -j160):
> - random build errors in binutils / gdb with messages like "No rule to
> make target 'po/BLD-POTFILES.in". I managed to reproduce something
> similar manually once, I noticed an empty Makefile; the build logs are
> of course difficult to read, so I couldn't figure out yet what could
> cause this.
> 
> - random build failures in gcc in fixincludes. I think this is a race
> condition because fixincludes is updated concurrently both from
> /fixincludes and $buillddir/fixincludes. Probably fixable in gcc
> Makefiles.
> 
> - I've seen other errors when building gcc like
> configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
> from libquadmath. I haven't investigated this yet.
> 
> I've read binutils' README-maintainer-mode, which contains a warning
> about distclean, but we don't use this: we start our builds from a
> scratch directory.
> 
> So... I'm wondering if there are some "official" guidelines about how
> to regenerate files, and/or use maintainer-mode?  Maybe I missed a
> "magic" make target (eg 'make autoreconf-all') that should be executed
> after configure and before 'make all'?
> 
> I've noticed that sourceware's buildbot has a small script
> "autoregen.py" which does not use the project's build system, but
> rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> Should we replicate that?
> 
> Thanks,
> 
> Christophe

There are other potential gotchas as well, such as the manual copying of the generated tm.texi back into the source repo due to relicensing.  Perhaps we haven't encountered that one because patches generally contain that duplicated output.

If we want a CI to work reliably, then perhaps we should reconsider our policy of stripping out regenerated code.  We have a number of developer practices, such as replying to an existing patch with an updated version that the CI can't handle easily (especially if the patch is part of a series), so there may be space for a discussion on how to work smarter.

My calendar says we have a toolchain office hours meeting today, perhaps this would be worth bringing up.

R.

Re: Help needed with maintainer-mode

[Christophe Lyon]

On Thu, 29 Feb 2024 at 11:41, Richard Earnshaw (lists)
<Richard.Earnshaw@arm.com> wrote:
>
> On 29/02/2024 10:22, Christophe Lyon via Gcc wrote:
> > Hi!
> >
> > Sorry for cross-posting, but I'm not sure the rules/guidelines are the
> > same in gcc vs binutils/gdb.
> >
> > TL;DR: are there some guidelines about how to use/enable maintainer-mode?
> >
> > In the context of the Linaro CI, I've been looking at enabling
> > maintainer-mode at configure time in our configurations where we test
> > patches before they are committed (aka "precommit CI", which relies on
> > patchwork).
> >
> > Indeed, auto-generated files are not part of patch submissions, and
> > when a patch implies regenerating some files before building, we
> > currently report wrong failures because we don't perform such updates.
> >
> > I hoped improving this would be as simple as adding
> > --enable-maintainer-mode when configuring, after making sure
> > autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
> > libtool and gettext seems OK).
> >
> > However, doing so triggered several problems, which look like race
> > conditions in the build system (we build at -j160):
> > - random build errors in binutils / gdb with messages like "No rule to
> > make target 'po/BLD-POTFILES.in". I managed to reproduce something
> > similar manually once, I noticed an empty Makefile; the build logs are
> > of course difficult to read, so I couldn't figure out yet what could
> > cause this.
> >
> > - random build failures in gcc in fixincludes. I think this is a race
> > condition because fixincludes is updated concurrently both from
> > /fixincludes and $buillddir/fixincludes. Probably fixable in gcc
> > Makefiles.
> >
> > - I've seen other errors when building gcc like
> > configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
> > from libquadmath. I haven't investigated this yet.
> >
> > I've read binutils' README-maintainer-mode, which contains a warning
> > about distclean, but we don't use this: we start our builds from a
> > scratch directory.
> >
> > So... I'm wondering if there are some "official" guidelines about how
> > to regenerate files, and/or use maintainer-mode?  Maybe I missed a
> > "magic" make target (eg 'make autoreconf-all') that should be executed
> > after configure and before 'make all'?
> >
> > I've noticed that sourceware's buildbot has a small script
> > "autoregen.py" which does not use the project's build system, but
> > rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> > Should we replicate that?
> >
> > Thanks,
> >
> > Christophe
>
> There are other potential gotchas as well, such as the manual copying of the generated tm.texi back into the source repo due to relicensing.  Perhaps we haven't encountered that one because patches generally contain that duplicated output.
>
It did happen a few weeks ago, with a patch that was updating the
target hooks IIRC.

> If we want a CI to work reliably, then perhaps we should reconsider our policy of stripping out regenerated code.  We have a number of developer practices, such as replying to an existing patch with an updated version that the CI can't handle easily (especially if the patch is part of a series), so there may be space for a discussion on how to work smarter.
>
Sure, there are many things we can improve in the current workflow to
make it more CI friendly ;-)
But I was only asking how maintainer-mode is supposed to be used, so
that I can replicate the process in CI.
I couldn't find any documentation :-)

Thanks,

Christophe

> My calendar says we have a toolchain office hours meeting today, perhaps this would be worth bringing up.
>
> R.
>

Re: Help needed with maintainer-mode

[Mark Wielaard]

Hi Christophe,

On Thu, Feb 29, 2024 at 11:22:33AM +0100, Christophe Lyon via Gcc wrote:
> I've noticed that sourceware's buildbot has a small script
> "autoregen.py" which does not use the project's build system, but
> rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> Should we replicate that?

That python script works across gcc/binutils/gdb:
https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py

It is installed into a container file that has the exact autoconf and
automake version needed to regenerate the autotool files:
https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools

And it was indeed done this way because that way the files are
regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).

It is run on all commits and warns if it detects a change in the
(checked in) generated files.
https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen

Cheers,

Mark

Re: Help needed with maintainer-mode

[Christophe Lyon]

On Thu, 29 Feb 2024 at 12:00, Mark Wielaard <mark@klomp.org> wrote:
>
> Hi Christophe,
>
> On Thu, Feb 29, 2024 at 11:22:33AM +0100, Christophe Lyon via Gcc wrote:
> > I've noticed that sourceware's buildbot has a small script
> > "autoregen.py" which does not use the project's build system, but
> > rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> > Should we replicate that?
>
> That python script works across gcc/binutils/gdb:
> https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
>
> It is installed into a container file that has the exact autoconf and
> automake version needed to regenerate the autotool files:
> https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
>
> And it was indeed done this way because that way the files are
> regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).

I see. So it is possibly incomplete, in the sense that it may lack
some of the steps that maintainer-mode would perform?
For instance, gas for aarch64 has some *opcodes*.c files that need
regenerating before committing. The regeneration step is enabled in
maintainer-mode, so I guess the autoregen bots on Sourceware would
miss a problem with these files?

Thanks,

Christophe

>
> It is run on all commits and warns if it detects a change in the
> (checked in) generated files.
> https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
> https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
>
> Cheers,
>
> Mark

Re: Help needed with maintainer-mode

[Mark Wielaard]

Hi Christophe,

On Thu, 2024-02-29 at 18:39 +0100, Christophe Lyon wrote:
> On Thu, 29 Feb 2024 at 12:00, Mark Wielaard <mark@klomp.org> wrote:
> > That python script works across gcc/binutils/gdb:
> > https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
> > 
> > It is installed into a container file that has the exact autoconf and
> > automake version needed to regenerate the autotool files:
> > https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
> > 
> > And it was indeed done this way because that way the files are
> > regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).
> 
> I see. So it is possibly incomplete, in the sense that it may lack
> some of the steps that maintainer-mode would perform?
> For instance, gas for aarch64 has some *opcodes*.c files that need
> regenerating before committing. The regeneration step is enabled in
> maintainer-mode, so I guess the autoregen bots on Sourceware would
> miss a problem with these files?

Yes, it is certainly incomplete. But it is done this way because it is
my understanding that even the gcc release maintainers do the
automake/autoconf invocations by hand instead of running with configure
--enable-maintainer-mode.

Note that another part that isn't caught at the moment are the
regeneration of the opt.urls files. There is a patch for that pending:
https://inbox.sourceware.org/buildbot/20231215005908.GC12145@gnu.wildebeest.org/

But that is waiting for the actual opt.urls to be regenerated correctly
first:
https://inbox.sourceware.org/gcc-patches/20240224174258.GD1353@gnu.wildebeest.org/
Ping David?

It would be nice to have all these "regeneration targets" in one script
that could be used by both the pre-commit and post-commit checkers.

Cheers,

Mark

Re: Help needed with maintainer-mode

[Christophe Lyon]

On Fri, 1 Mar 2024 at 14:08, Mark Wielaard <mark@klomp.org> wrote:
>
> Hi Christophe,
>
> On Thu, 2024-02-29 at 18:39 +0100, Christophe Lyon wrote:
> > On Thu, 29 Feb 2024 at 12:00, Mark Wielaard <mark@klomp.org> wrote:
> > > That python script works across gcc/binutils/gdb:
> > > https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
> > >
> > > It is installed into a container file that has the exact autoconf and
> > > automake version needed to regenerate the autotool files:
> > > https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
> > >
> > > And it was indeed done this way because that way the files are
> > > regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).
> >
> > I see. So it is possibly incomplete, in the sense that it may lack
> > some of the steps that maintainer-mode would perform?
> > For instance, gas for aarch64 has some *opcodes*.c files that need
> > regenerating before committing. The regeneration step is enabled in
> > maintainer-mode, so I guess the autoregen bots on Sourceware would
> > miss a problem with these files?
>
> Yes, it is certainly incomplete. But it is done this way because it is
> my understanding that even the gcc release maintainers do the
> automake/autoconf invocations by hand instead of running with configure
> --enable-maintainer-mode.

Indeed, I've just discovered that earlier today :-)

>
> Note that another part that isn't caught at the moment are the
> regeneration of the opt.urls files. There is a patch for that pending:
Indeed. I hadn't thought of it either. And just noticed it requires
the D frontend, which we don't build in CI.

> https://inbox.sourceware.org/buildbot/20231215005908.GC12145@gnu.wildebeest.org/
>
> But that is waiting for the actual opt.urls to be regenerated correctly
> first:
> https://inbox.sourceware.org/gcc-patches/20240224174258.GD1353@gnu.wildebeest.org/
> Ping David?
>
> It would be nice to have all these "regeneration targets" in one script
> that could be used by both the pre-commit and post-commit checkers.
>
Agreed.

> Cheers,
>
> Mark

Re: Help needed with maintainer-mode

[Christophe Lyon]

On Fri, 1 Mar 2024 at 14:08, Mark Wielaard <mark@klomp.org> wrote:
>
> Hi Christophe,
>
> On Thu, 2024-02-29 at 18:39 +0100, Christophe Lyon wrote:
> > On Thu, 29 Feb 2024 at 12:00, Mark Wielaard <mark@klomp.org> wrote:
> > > That python script works across gcc/binutils/gdb:
> > > https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
> > >
> > > It is installed into a container file that has the exact autoconf and
> > > automake version needed to regenerate the autotool files:
> > > https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
> > >
> > > And it was indeed done this way because that way the files are
> > > regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).
> >
> > I see. So it is possibly incomplete, in the sense that it may lack
> > some of the steps that maintainer-mode would perform?
> > For instance, gas for aarch64 has some *opcodes*.c files that need
> > regenerating before committing. The regeneration step is enabled in
> > maintainer-mode, so I guess the autoregen bots on Sourceware would
> > miss a problem with these files?
>
> Yes, it is certainly incomplete. But it is done this way because it is
> my understanding that even the gcc release maintainers do the
> automake/autoconf invocations by hand instead of running with configure
> --enable-maintainer-mode.

After a discussion on IRC, I read
https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
which basically says "run autoreconf in every dir where there is a
configure script"
but this is not exactly what autoregen.py is doing. IIRC it is based
on a script from Martin Liska, do you know/remember why it follows a
different process?

Thanks,

Christophe

>
> Note that another part that isn't caught at the moment are the
> regeneration of the opt.urls files. There is a patch for that pending:
> https://inbox.sourceware.org/buildbot/20231215005908.GC12145@gnu.wildebeest.org/
>
> But that is waiting for the actual opt.urls to be regenerated correctly
> first:
> https://inbox.sourceware.org/gcc-patches/20240224174258.GD1353@gnu.wildebeest.org/
> Ping David?
>
> It would be nice to have all these "regeneration targets" in one script
> that could be used by both the pre-commit and post-commit checkers.
>
> Cheers,
>
> Mark

Re: Help needed with maintainer-mode

[Mark Wielaard]

Hi Christophe,

On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> > > > And it was indeed done this way because that way the files are
> > > > regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).
> > >
> > > I see. So it is possibly incomplete, in the sense that it may lack
> > > some of the steps that maintainer-mode would perform?
> > > For instance, gas for aarch64 has some *opcodes*.c files that need
> > > regenerating before committing. The regeneration step is enabled in
> > > maintainer-mode, so I guess the autoregen bots on Sourceware would
> > > miss a problem with these files?
> >
> > Yes, it is certainly incomplete. But it is done this way because it is
> > my understanding that even the gcc release maintainers do the
> > automake/autoconf invocations by hand instead of running with configure
> > --enable-maintainer-mode.
> 
> After a discussion on IRC, I read
> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> which basically says "run autoreconf in every dir where there is a
> configure script"
> but this is not exactly what autoregen.py is doing. IIRC it is based
> on a script from Martin Liska, do you know/remember why it follows a
> different process?

CCing Sam and Arsen who helped refine the autoregen.py script, who
might remember more details. We wanted a script that worked for both
gcc and binutils-gdb. And as far as I know autoreconf simply didn't
work in all directories. We also needed to skip some directories that
did contain a configure script, but that were imported (gotools,
readline, minizip).

Cheers,

Mark

Re: Help needed with maintainer-mode

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]

Mark Wielaard <mark@klomp.org> writes:

> Hi Christophe,

Hi Mark, Christophe, et. al,

>
> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>> > > > And it was indeed done this way because that way the files are
>> > > > regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).
>> > >
>> > > I see. So it is possibly incomplete, in the sense that it may lack
>> > > some of the steps that maintainer-mode would perform?
>> > > For instance, gas for aarch64 has some *opcodes*.c files that need
>> > > regenerating before committing. The regeneration step is enabled in
>> > > maintainer-mode, so I guess the autoregen bots on Sourceware would
>> > > miss a problem with these files?
>> >
>> > Yes, it is certainly incomplete. But it is done this way because it is
>> > my understanding that even the gcc release maintainers do the
>> > automake/autoconf invocations by hand instead of running with configure
>> > --enable-maintainer-mode.
>> 
>> After a discussion on IRC, I read
>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>> which basically says "run autoreconf in every dir where there is a
>> configure script"
>> but this is not exactly what autoregen.py is doing. IIRC it is based
>> on a script from Martin Liska, do you know/remember why it follows a
>> different process?
>
> CCing Sam and Arsen who helped refine the autoregen.py script, who
> might remember more details. We wanted a script that worked for both
> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> work in all directories. We also needed to skip some directories that
> did contain a configure script, but that were imported (gotools,
> readline, minizip).

What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
for AC_CONFIG_SUBDIRS.

Right now, the current situation is janky and it's nowhere near
idiomatic autotools usage. It is not a comfortable experience
interacting with it even as someone who is familiar and happy with using
autotools otherwise.

I didn't yet play with maintainer-mode myself but I also didn't see much
point given I knew of more fundamental problems like this.

[0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/

>
> Cheers,
>
> Mark

best,
sam

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: Help needed with maintainer-mode

[Thomas Schwinge]

Hi!

On 2024-03-04T00:30:05+0000, Sam James <sam@gentoo.org> wrote:
> Mark Wielaard <mark@klomp.org> writes:
>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>>> [...], I read
>>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>>> which basically says "run autoreconf in every dir where there is a
>>> configure script"
>>> but this is not exactly what autoregen.py is doing. IIRC it is based
>>> on a script from Martin Liska, do you know/remember why it follows a
>>> different process?
>>
>> CCing Sam and Arsen who helped refine the autoregen.py script, who
>> might remember more details. We wanted a script that worked for both
>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
>> work in all directories. We also needed to skip some directories that
>> did contain a configure script, but that were imported (gotools,
>> readline, minizip).
>
> What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
> for AC_CONFIG_SUBDIRS.

Let me allocate some time this week to get that one completed.

> Right now, the current situation is janky and it's nowhere near
> idiomatic autotools usage. It is not a comfortable experience
> interacting with it even as someone who is familiar and happy with using
> autotools otherwise.
>
> I didn't yet play with maintainer-mode myself but I also didn't see much
> point given I knew of more fundamental problems like this.
>
> [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/


Grüße
 Thomas

Re: Help needed with maintainer-mode

[Christophe Lyon]

Hi!

On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge <tschwinge@baylibre.com> wrote:
>
> Hi!
>
> On 2024-03-04T00:30:05+0000, Sam James <sam@gentoo.org> wrote:
> > Mark Wielaard <mark@klomp.org> writes:
> >> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> >>> [...], I read
> >>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> >>> which basically says "run autoreconf in every dir where there is a
> >>> configure script"
> >>> but this is not exactly what autoregen.py is doing. IIRC it is based
> >>> on a script from Martin Liska, do you know/remember why it follows a
> >>> different process?
> >>
> >> CCing Sam and Arsen who helped refine the autoregen.py script, who
> >> might remember more details. We wanted a script that worked for both
> >> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> >> work in all directories. We also needed to skip some directories that
> >> did contain a configure script, but that were imported (gotools,
> >> readline, minizip).
> >
> > What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
> > for AC_CONFIG_SUBDIRS.
>
> Let me allocate some time this week to get that one completed.
>
> > Right now, the current situation is janky and it's nowhere near
> > idiomatic autotools usage. It is not a comfortable experience
> > interacting with it even as someone who is familiar and happy with using
> > autotools otherwise.
> >
> > I didn't yet play with maintainer-mode myself but I also didn't see much
> > point given I knew of more fundamental problems like this.
> >
> > [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/
>

Thanks for the background. I didn't follow that discussion at that time :-)

So... I was confused because I noticed many warnings when doing a simple
find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
autoreconf -f $d && echo $d; done
as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration

Then I tried with autoregen.py, and saw the same.... and now just
checked Sourceware's bot logs and saw the same numerous warnings at
least in GCC (didn't check binutils yet). Looks like this is
"expected" ....

I started looking at auto-regenerating these files in our CI a couple
of weeks ago, after we received several "complaints" from contributors
saying that our precommit CI was useless / bothering since it didn't
regenerate files, leading to false alarms.
But now I'm wondering how such contributors regenerate the files
impacted by their patches before committing, they probably just
regenerate things in their subdir of interest, not noticing the whole
picture :-(

As a first step, we can probably use autoregen.py too, and declare
maintainer-mode broken. However, I do notice that besides the rules
about regenerating configure/Makefile.in/..., maintainer-mode is also
used to update some files.
In gcc:
fixincludes: fixincl.x
libffi: doc/version.texi
libgfortran: some stuff :-)
libiberty: functions.texi

in binutils/bfd:
gdb/sim
bfd/po/SRC-POTFILES.in
bfd/po/BLD-POTFILES.in
bfd/bfd-in2.h
bfd/libbfd.h
bfd/libcoff.h
binutils/po/POTFILES.in
gas/po/POTFILES.in
opcodes/i386*.h
gdb/copying.c
gdb/data-directory/*.xml
gold/po/POTFILES.in
gprof/po/POTFILES.in
gfprofng/doc/version.texi
ld/po/SRC-POTFILES.in
ld/po/BLD-POTFILES.in
ld: ldgram/ldlex... and all emulation sources
libiberty/functions.texi
opcodes/po/POTFILES.in
opcodes/aarch64-{asm,dis,opc}-2.c
opcodes/ia64 msp430 rl78 rx z8k decoders

How are these files "normally" expected to be updated? Do people just
manually uncomment the corresponding maintainer lines in the Makefiles
and update manually?   In particular we hit issues several times with
files under opcodes, that we don't regenerate currently. Maybe we
could build binutils in maintainer-mode at -j1 but, well....

README-maintainer-mode in binutils/gdb only mentions a problem with
'make distclean' and maintainer mode
binutils/README-how-to-make-a-release indicates to use
--enable-maintainer-mode, and the sample 'make' invocations do not
include any -j flag, is that an indication that only -j1 is supposed
to work?
Similarly, the src-release.sh script does not use -j.

Thanks,

Christophe

>
> Grüße
>  Thomas

Re: Help needed with maintainer-mode

[Jonathan Wakely]

On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
>
> Hi!
>
> On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge <tschwinge@baylibre.com> wrote:
> >
> > Hi!
> >
> > On 2024-03-04T00:30:05+0000, Sam James <sam@gentoo.org> wrote:
> > > Mark Wielaard <mark@klomp.org> writes:
> > >> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> > >>> [...], I read
> > >>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> > >>> which basically says "run autoreconf in every dir where there is a
> > >>> configure script"
> > >>> but this is not exactly what autoregen.py is doing. IIRC it is based
> > >>> on a script from Martin Liska, do you know/remember why it follows a
> > >>> different process?
> > >>
> > >> CCing Sam and Arsen who helped refine the autoregen.py script, who
> > >> might remember more details. We wanted a script that worked for both
> > >> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> > >> work in all directories. We also needed to skip some directories that
> > >> did contain a configure script, but that were imported (gotools,
> > >> readline, minizip).
> > >
> > > What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
> > > for AC_CONFIG_SUBDIRS.
> >
> > Let me allocate some time this week to get that one completed.
> >
> > > Right now, the current situation is janky and it's nowhere near
> > > idiomatic autotools usage. It is not a comfortable experience
> > > interacting with it even as someone who is familiar and happy with using
> > > autotools otherwise.
> > >
> > > I didn't yet play with maintainer-mode myself but I also didn't see much
> > > point given I knew of more fundamental problems like this.
> > >
> > > [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/
> >
>
> Thanks for the background. I didn't follow that discussion at that time :-)
>
> So... I was confused because I noticed many warnings when doing a simple
> find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
> autoreconf -f $d && echo $d; done
> as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>
> Then I tried with autoregen.py, and saw the same.... and now just
> checked Sourceware's bot logs and saw the same numerous warnings at
> least in GCC (didn't check binutils yet). Looks like this is
> "expected" ....
>
> I started looking at auto-regenerating these files in our CI a couple
> of weeks ago, after we received several "complaints" from contributors
> saying that our precommit CI was useless / bothering since it didn't
> regenerate files, leading to false alarms.
> But now I'm wondering how such contributors regenerate the files
> impacted by their patches before committing, they probably just
> regenerate things in their subdir of interest, not noticing the whole
> picture :-(
>
> As a first step, we can probably use autoregen.py too, and declare
> maintainer-mode broken. However, I do notice that besides the rules
> about regenerating configure/Makefile.in/..., maintainer-mode is also
> used to update some files.
> In gcc:
> fixincludes: fixincl.x
> libffi: doc/version.texi
> libgfortran: some stuff :-)
> libiberty: functions.texi

My recently proposed patch adds the first of those to gcc_update, the
other should be done too.
https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html



>
> in binutils/bfd:
> gdb/sim
> bfd/po/SRC-POTFILES.in
> bfd/po/BLD-POTFILES.in
> bfd/bfd-in2.h
> bfd/libbfd.h
> bfd/libcoff.h
> binutils/po/POTFILES.in
> gas/po/POTFILES.in
> opcodes/i386*.h
> gdb/copying.c
> gdb/data-directory/*.xml
> gold/po/POTFILES.in
> gprof/po/POTFILES.in
> gfprofng/doc/version.texi
> ld/po/SRC-POTFILES.in
> ld/po/BLD-POTFILES.in
> ld: ldgram/ldlex... and all emulation sources
> libiberty/functions.texi
> opcodes/po/POTFILES.in
> opcodes/aarch64-{asm,dis,opc}-2.c
> opcodes/ia64 msp430 rl78 rx z8k decoders
>
> How are these files "normally" expected to be updated? Do people just
> manually uncomment the corresponding maintainer lines in the Makefiles
> and update manually?   In particular we hit issues several times with
> files under opcodes, that we don't regenerate currently. Maybe we
> could build binutils in maintainer-mode at -j1 but, well....
>
> README-maintainer-mode in binutils/gdb only mentions a problem with
> 'make distclean' and maintainer mode
> binutils/README-how-to-make-a-release indicates to use
> --enable-maintainer-mode, and the sample 'make' invocations do not
> include any -j flag, is that an indication that only -j1 is supposed
> to work?
> Similarly, the src-release.sh script does not use -j.
>
> Thanks,
>
> Christophe
>
> >
> > Grüße
> >  Thomas

Re: Help needed with maintainer-mode

[Christophe Lyon]

On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely <jwakely.gcc@gmail.com> wrote:
>
> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
> >
> > Hi!
> >
> > On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge <tschwinge@baylibre.com> wrote:
> > >
> > > Hi!
> > >
> > > On 2024-03-04T00:30:05+0000, Sam James <sam@gentoo.org> wrote:
> > > > Mark Wielaard <mark@klomp.org> writes:
> > > >> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> > > >>> [...], I read
> > > >>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> > > >>> which basically says "run autoreconf in every dir where there is a
> > > >>> configure script"
> > > >>> but this is not exactly what autoregen.py is doing. IIRC it is based
> > > >>> on a script from Martin Liska, do you know/remember why it follows a
> > > >>> different process?
> > > >>
> > > >> CCing Sam and Arsen who helped refine the autoregen.py script, who
> > > >> might remember more details. We wanted a script that worked for both
> > > >> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> > > >> work in all directories. We also needed to skip some directories that
> > > >> did contain a configure script, but that were imported (gotools,
> > > >> readline, minizip).
> > > >
> > > > What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
> > > > for AC_CONFIG_SUBDIRS.
> > >
> > > Let me allocate some time this week to get that one completed.
> > >
> > > > Right now, the current situation is janky and it's nowhere near
> > > > idiomatic autotools usage. It is not a comfortable experience
> > > > interacting with it even as someone who is familiar and happy with using
> > > > autotools otherwise.
> > > >
> > > > I didn't yet play with maintainer-mode myself but I also didn't see much
> > > > point given I knew of more fundamental problems like this.
> > > >
> > > > [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/
> > >
> >
> > Thanks for the background. I didn't follow that discussion at that time :-)
> >
> > So... I was confused because I noticed many warnings when doing a simple
> > find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
> > autoreconf -f $d && echo $d; done
> > as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
> >
> > Then I tried with autoregen.py, and saw the same.... and now just
> > checked Sourceware's bot logs and saw the same numerous warnings at
> > least in GCC (didn't check binutils yet). Looks like this is
> > "expected" ....
> >
> > I started looking at auto-regenerating these files in our CI a couple
> > of weeks ago, after we received several "complaints" from contributors
> > saying that our precommit CI was useless / bothering since it didn't
> > regenerate files, leading to false alarms.
> > But now I'm wondering how such contributors regenerate the files
> > impacted by their patches before committing, they probably just
> > regenerate things in their subdir of interest, not noticing the whole
> > picture :-(
> >
> > As a first step, we can probably use autoregen.py too, and declare
> > maintainer-mode broken. However, I do notice that besides the rules
> > about regenerating configure/Makefile.in/..., maintainer-mode is also
> > used to update some files.
> > In gcc:
> > fixincludes: fixincl.x
> > libffi: doc/version.texi
> > libgfortran: some stuff :-)
> > libiberty: functions.texi
>
> My recently proposed patch adds the first of those to gcc_update, the
> other should be done too.
> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html
>

This script touches files such that they appear more recent than their
dependencies,
so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
For auto* files, this is "fine" as we can run autoreconf or
autoregen.py before starting configure+build, but what about other
files?
For instance, if we have to test a patch which implies changes to
fixincludes/fixincl.x, how should we proceed?
1- git checkout (with possibly "wrong" timestamps)
2- apply patch-to-test
3- contrib/gcc_update -t
4- configure --enable-maintainer-mode

I guess --enable-maintainer-mode would be largely (if not completely)
disabled by step 3 above?
And we should probably swap steps 2 and 3, so that the effects of
patch-to-test are handled by make?

Thanks,

Christophe

>
>
> >
> > in binutils/bfd:
> > gdb/sim
> > bfd/po/SRC-POTFILES.in
> > bfd/po/BLD-POTFILES.in
> > bfd/bfd-in2.h
> > bfd/libbfd.h
> > bfd/libcoff.h
> > binutils/po/POTFILES.in
> > gas/po/POTFILES.in
> > opcodes/i386*.h
> > gdb/copying.c
> > gdb/data-directory/*.xml
> > gold/po/POTFILES.in
> > gprof/po/POTFILES.in
> > gfprofng/doc/version.texi
> > ld/po/SRC-POTFILES.in
> > ld/po/BLD-POTFILES.in
> > ld: ldgram/ldlex... and all emulation sources
> > libiberty/functions.texi
> > opcodes/po/POTFILES.in
> > opcodes/aarch64-{asm,dis,opc}-2.c
> > opcodes/ia64 msp430 rl78 rx z8k decoders
> >
> > How are these files "normally" expected to be updated? Do people just
> > manually uncomment the corresponding maintainer lines in the Makefiles
> > and update manually?   In particular we hit issues several times with
> > files under opcodes, that we don't regenerate currently. Maybe we
> > could build binutils in maintainer-mode at -j1 but, well....
> >
> > README-maintainer-mode in binutils/gdb only mentions a problem with
> > 'make distclean' and maintainer mode
> > binutils/README-how-to-make-a-release indicates to use
> > --enable-maintainer-mode, and the sample 'make' invocations do not
> > include any -j flag, is that an indication that only -j1 is supposed
> > to work?
> > Similarly, the src-release.sh script does not use -j.
> >
> > Thanks,
> >
> > Christophe
> >
> > >
> > > Grüße
> > >  Thomas

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists)]

On 04/03/2024 14:46, Christophe Lyon via Gcc wrote:
> On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely <jwakely.gcc@gmail.com> wrote:
>>
>> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
>>>
>>> Hi!
>>>
>>> On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge <tschwinge@baylibre.com> wrote:
>>>>
>>>> Hi!
>>>>
>>>> On 2024-03-04T00:30:05+0000, Sam James <sam@gentoo.org> wrote:
>>>>> Mark Wielaard <mark@klomp.org> writes:
>>>>>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>>>>>>> [...], I read
>>>>>>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>>>>>>> which basically says "run autoreconf in every dir where there is a
>>>>>>> configure script"
>>>>>>> but this is not exactly what autoregen.py is doing. IIRC it is based
>>>>>>> on a script from Martin Liska, do you know/remember why it follows a
>>>>>>> different process?
>>>>>>
>>>>>> CCing Sam and Arsen who helped refine the autoregen.py script, who
>>>>>> might remember more details. We wanted a script that worked for both
>>>>>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
>>>>>> work in all directories. We also needed to skip some directories that
>>>>>> did contain a configure script, but that were imported (gotools,
>>>>>> readline, minizip).
>>>>>
>>>>> What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
>>>>> for AC_CONFIG_SUBDIRS.
>>>>
>>>> Let me allocate some time this week to get that one completed.
>>>>
>>>>> Right now, the current situation is janky and it's nowhere near
>>>>> idiomatic autotools usage. It is not a comfortable experience
>>>>> interacting with it even as someone who is familiar and happy with using
>>>>> autotools otherwise.
>>>>>
>>>>> I didn't yet play with maintainer-mode myself but I also didn't see much
>>>>> point given I knew of more fundamental problems like this.
>>>>>
>>>>> [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/
>>>>
>>>
>>> Thanks for the background. I didn't follow that discussion at that time :-)
>>>
>>> So... I was confused because I noticed many warnings when doing a simple
>>> find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
>>> autoreconf -f $d && echo $d; done
>>> as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration
>>>
>>> Then I tried with autoregen.py, and saw the same.... and now just
>>> checked Sourceware's bot logs and saw the same numerous warnings at
>>> least in GCC (didn't check binutils yet). Looks like this is
>>> "expected" ....
>>>
>>> I started looking at auto-regenerating these files in our CI a couple
>>> of weeks ago, after we received several "complaints" from contributors
>>> saying that our precommit CI was useless / bothering since it didn't
>>> regenerate files, leading to false alarms.
>>> But now I'm wondering how such contributors regenerate the files
>>> impacted by their patches before committing, they probably just
>>> regenerate things in their subdir of interest, not noticing the whole
>>> picture :-(
>>>
>>> As a first step, we can probably use autoregen.py too, and declare
>>> maintainer-mode broken. However, I do notice that besides the rules
>>> about regenerating configure/Makefile.in/..., maintainer-mode is also
>>> used to update some files.
>>> In gcc:
>>> fixincludes: fixincl.x
>>> libffi: doc/version.texi
>>> libgfortran: some stuff :-)
>>> libiberty: functions.texi
>>
>> My recently proposed patch adds the first of those to gcc_update, the
>> other should be done too.
>> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html
>>
> 
> This script touches files such that they appear more recent than their
> dependencies,
> so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
> For auto* files, this is "fine" as we can run autoreconf or
> autoregen.py before starting configure+build, but what about other
> files?
> For instance, if we have to test a patch which implies changes to
> fixincludes/fixincl.x, how should we proceed?
> 1- git checkout (with possibly "wrong" timestamps)
> 2- apply patch-to-test
> 3- contrib/gcc_update -t
> 4- configure --enable-maintainer-mode

If you ran

git reset --hard master // restore state to 'master'
contrib/gcc_update // pull latest code

then anything coming from upstream will be touched automatically.  You really don't want to re-touch the files after patching unless you're sure they've all been patched correctly, it will break if there's anything regenerated that's missing.

R.

> 
> I guess --enable-maintainer-mode would be largely (if not completely)
> disabled by step 3 above?
> And we should probably swap steps 2 and 3, so that the effects of
> patch-to-test are handled by make?
> 
> Thanks,
> 
> Christophe
> 
>>
>>
>>>
>>> in binutils/bfd:
>>> gdb/sim
>>> bfd/po/SRC-POTFILES.in
>>> bfd/po/BLD-POTFILES.in
>>> bfd/bfd-in2.h
>>> bfd/libbfd.h
>>> bfd/libcoff.h
>>> binutils/po/POTFILES.in
>>> gas/po/POTFILES.in
>>> opcodes/i386*.h
>>> gdb/copying.c
>>> gdb/data-directory/*.xml
>>> gold/po/POTFILES.in
>>> gprof/po/POTFILES.in
>>> gfprofng/doc/version.texi
>>> ld/po/SRC-POTFILES.in
>>> ld/po/BLD-POTFILES.in
>>> ld: ldgram/ldlex... and all emulation sources
>>> libiberty/functions.texi
>>> opcodes/po/POTFILES.in
>>> opcodes/aarch64-{asm,dis,opc}-2.c
>>> opcodes/ia64 msp430 rl78 rx z8k decoders
>>>
>>> How are these files "normally" expected to be updated? Do people just
>>> manually uncomment the corresponding maintainer lines in the Makefiles
>>> and update manually?   In particular we hit issues several times with
>>> files under opcodes, that we don't regenerate currently. Maybe we
>>> could build binutils in maintainer-mode at -j1 but, well....
>>>
>>> README-maintainer-mode in binutils/gdb only mentions a problem with
>>> 'make distclean' and maintainer mode
>>> binutils/README-how-to-make-a-release indicates to use
>>> --enable-maintainer-mode, and the sample 'make' invocations do not
>>> include any -j flag, is that an indication that only -j1 is supposed
>>> to work?
>>> Similarly, the src-release.sh script does not use -j.
>>>
>>> Thanks,
>>>
>>> Christophe
>>>
>>>>
>>>> Grüße
>>>>  Thomas

Re: Help needed with maintainer-mode

[Richard Earnshaw]

On 04/03/2024 15:36, Richard Earnshaw (lists) wrote:
> On 04/03/2024 14:46, Christophe Lyon via Gcc wrote:
>> On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely <jwakely.gcc@gmail.com> wrote:
>>>
>>> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
>>>>
>>>> Hi!
>>>>
>>>> On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge <tschwinge@baylibre.com> wrote:
>>>>>
>>>>> Hi!
>>>>>
>>>>> On 2024-03-04T00:30:05+0000, Sam James <sam@gentoo.org> wrote:
>>>>>> Mark Wielaard <mark@klomp.org> writes:
>>>>>>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>>>>>>>> [...], I read
>>>>>>>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration>
>>>>>>>> which basically says "run autoreconf in every dir where there is a
>>>>>>>> configure script"
>>>>>>>> but this is not exactly what autoregen.py is doing. IIRC it is based
>>>>>>>> on a script from Martin Liska, do you know/remember why it follows a
>>>>>>>> different process?
>>>>>>>
>>>>>>> CCing Sam and Arsen who helped refine the autoregen.py script, who
>>>>>>> might remember more details. We wanted a script that worked for both
>>>>>>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
>>>>>>> work in all directories. We also needed to skip some directories that
>>>>>>> did contain a configure script, but that were imported (gotools,
>>>>>>> readline, minizip).
>>>>>>
>>>>>> What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
>>>>>> for AC_CONFIG_SUBDIRS.
>>>>>
>>>>> Let me allocate some time this week to get that one completed.
>>>>>
>>>>>> Right now, the current situation is janky and it's nowhere near
>>>>>> idiomatic autotools usage. It is not a comfortable experience
>>>>>> interacting with it even as someone who is familiar and happy with using
>>>>>> autotools otherwise.
>>>>>>
>>>>>> I didn't yet play with maintainer-mode myself but I also didn't see much
>>>>>> point given I knew of more fundamental problems like this.
>>>>>>
>>>>>> [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/ <https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/>
>>>>>
>>>>
>>>> Thanks for the background. I didn't follow that discussion at that time :-)
>>>>
>>>> So... I was confused because I noticed many warnings when doing a simple
>>>> find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
>>>> autoreconf -f $d && echo $d; done
>>>> as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration>
>>>>
>>>> Then I tried with autoregen.py, and saw the same.... and now just
>>>> checked Sourceware's bot logs and saw the same numerous warnings at
>>>> least in GCC (didn't check binutils yet). Looks like this is
>>>> "expected" ....
>>>>
>>>> I started looking at auto-regenerating these files in our CI a couple
>>>> of weeks ago, after we received several "complaints" from contributors
>>>> saying that our precommit CI was useless / bothering since it didn't
>>>> regenerate files, leading to false alarms.
>>>> But now I'm wondering how such contributors regenerate the files
>>>> impacted by their patches before committing, they probably just
>>>> regenerate things in their subdir of interest, not noticing the whole
>>>> picture :-(
>>>>
>>>> As a first step, we can probably use autoregen.py too, and declare
>>>> maintainer-mode broken. However, I do notice that besides the rules
>>>> about regenerating configure/Makefile.in/..., maintainer-mode is also
>>>> used to update some files.
>>>> In gcc:
>>>> fixincludes: fixincl.x
>>>> libffi: doc/version.texi
>>>> libgfortran: some stuff :-)
>>>> libiberty: functions.texi
>>>
>>> My recently proposed patch adds the first of those to gcc_update, the
>>> other should be done too.
>>> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html <https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html>
>>>
>> 
>> This script touches files such that they appear more recent than their
>> dependencies,
>> so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
>> For auto* files, this is "fine" as we can run autoreconf or
>> autoregen.py before starting configure+build, but what about other
>> files?
>> For instance, if we have to test a patch which implies changes to
>> fixincludes/fixincl.x, how should we proceed?
>> 1- git checkout (with possibly "wrong" timestamps)
>> 2- apply patch-to-test
>> 3- contrib/gcc_update -t
>> 4- configure --enable-maintainer-mode
> 
> If you ran
> 
> git reset --hard master // restore state to 'master'
> contrib/gcc_update // pull latest code
> 
> then anything coming from upstream will be touched automatically.  You really don't want to re-touch the files after patching unless you're sure they've all been patched correctly, it will break if there's anything regenerated that's missing.
> 
> R.

Alternatively, if you did 

git reset --hard master // restore state to 'master'
contrib/gcc_update // pull latest code
patch
contrib/gcc_update -t

and the second command touched files that weren't listed in the patches, then you'd know that there was something missing from the patch set.  You'd probably still need to go back to the pristine trunk version though, to get the state into something that could be maintainer-mode regenerated.

R.

> 
>> 
>> I guess --enable-maintainer-mode would be largely (if not completely)
>> disabled by step 3 above?
>> And we should probably swap steps 2 and 3, so that the effects of
>> patch-to-test are handled by make?
>> 
>> Thanks,
>> 
>> Christophe
>> 
>>>
>>>
>>>>
>>>> in binutils/bfd:
>>>> gdb/sim
>>>> bfd/po/SRC-POTFILES.in
>>>> bfd/po/BLD-POTFILES.in
>>>> bfd/bfd-in2.h
>>>> bfd/libbfd.h
>>>> bfd/libcoff.h
>>>> binutils/po/POTFILES.in
>>>> gas/po/POTFILES.in
>>>> opcodes/i386*.h
>>>> gdb/copying.c
>>>> gdb/data-directory/*.xml
>>>> gold/po/POTFILES.in
>>>> gprof/po/POTFILES.in
>>>> gfprofng/doc/version.texi
>>>> ld/po/SRC-POTFILES.in
>>>> ld/po/BLD-POTFILES.in
>>>> ld: ldgram/ldlex... and all emulation sources
>>>> libiberty/functions.texi
>>>> opcodes/po/POTFILES.in
>>>> opcodes/aarch64-{asm,dis,opc}-2.c
>>>> opcodes/ia64 msp430 rl78 rx z8k decoders
>>>>
>>>> How are these files "normally" expected to be updated? Do people just
>>>> manually uncomment the corresponding maintainer lines in the Makefiles
>>>> and update manually?   In particular we hit issues several times with
>>>> files under opcodes, that we don't regenerate currently. Maybe we
>>>> could build binutils in maintainer-mode at -j1 but, well....
>>>>
>>>> README-maintainer-mode in binutils/gdb only mentions a problem with
>>>> 'make distclean' and maintainer mode
>>>> binutils/README-how-to-make-a-release indicates to use
>>>> --enable-maintainer-mode, and the sample 'make' invocations do not
>>>> include any -j flag, is that an indication that only -j1 is supposed
>>>> to work?
>>>> Similarly, the src-release.sh script does not use -j.
>>>>
>>>> Thanks,
>>>>
>>>> Christophe
>>>>
>>>>>
>>>>> Grüße
>>>>>  Thomas
> 

Re: Help needed with maintainer-mode

[Christophe Lyon]

On Mon, 4 Mar 2024 at 16:41, Richard Earnshaw <Richard.Earnshaw@arm.com> wrote:
>
>
>
> On 04/03/2024 15:36, Richard Earnshaw (lists) wrote:
> > On 04/03/2024 14:46, Christophe Lyon via Gcc wrote:
> >> On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely <jwakely.gcc@gmail.com> wrote:
> >>>
> >>> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
> >>>>
> >>>> Hi!
> >>>>
> >>>> On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge <tschwinge@baylibre.com> wrote:
> >>>>>
> >>>>> Hi!
> >>>>>
> >>>>> On 2024-03-04T00:30:05+0000, Sam James <sam@gentoo.org> wrote:
> >>>>>> Mark Wielaard <mark@klomp.org> writes:
> >>>>>>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
> >>>>>>>> [...], I read
> >>>>>>>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration>
> >>>>>>>> which basically says "run autoreconf in every dir where there is a
> >>>>>>>> configure script"
> >>>>>>>> but this is not exactly what autoregen.py is doing. IIRC it is based
> >>>>>>>> on a script from Martin Liska, do you know/remember why it follows a
> >>>>>>>> different process?
> >>>>>>>
> >>>>>>> CCing Sam and Arsen who helped refine the autoregen.py script, who
> >>>>>>> might remember more details. We wanted a script that worked for both
> >>>>>>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
> >>>>>>> work in all directories. We also needed to skip some directories that
> >>>>>>> did contain a configure script, but that were imported (gotools,
> >>>>>>> readline, minizip).
> >>>>>>
> >>>>>> What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
> >>>>>> for AC_CONFIG_SUBDIRS.
> >>>>>
> >>>>> Let me allocate some time this week to get that one completed.
> >>>>>
> >>>>>> Right now, the current situation is janky and it's nowhere near
> >>>>>> idiomatic autotools usage. It is not a comfortable experience
> >>>>>> interacting with it even as someone who is familiar and happy with using
> >>>>>> autotools otherwise.
> >>>>>>
> >>>>>> I didn't yet play with maintainer-mode myself but I also didn't see much
> >>>>>> point given I knew of more fundamental problems like this.
> >>>>>>
> >>>>>> [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/ <https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/>
> >>>>>
> >>>>
> >>>> Thanks for the background. I didn't follow that discussion at that time :-)
> >>>>
> >>>> So... I was confused because I noticed many warnings when doing a simple
> >>>> find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
> >>>> autoreconf -f $d && echo $d; done
> >>>> as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration>
> >>>>
> >>>> Then I tried with autoregen.py, and saw the same.... and now just
> >>>> checked Sourceware's bot logs and saw the same numerous warnings at
> >>>> least in GCC (didn't check binutils yet). Looks like this is
> >>>> "expected" ....
> >>>>
> >>>> I started looking at auto-regenerating these files in our CI a couple
> >>>> of weeks ago, after we received several "complaints" from contributors
> >>>> saying that our precommit CI was useless / bothering since it didn't
> >>>> regenerate files, leading to false alarms.
> >>>> But now I'm wondering how such contributors regenerate the files
> >>>> impacted by their patches before committing, they probably just
> >>>> regenerate things in their subdir of interest, not noticing the whole
> >>>> picture :-(
> >>>>
> >>>> As a first step, we can probably use autoregen.py too, and declare
> >>>> maintainer-mode broken. However, I do notice that besides the rules
> >>>> about regenerating configure/Makefile.in/..., maintainer-mode is also
> >>>> used to update some files.
> >>>> In gcc:
> >>>> fixincludes: fixincl.x
> >>>> libffi: doc/version.texi
> >>>> libgfortran: some stuff :-)
> >>>> libiberty: functions.texi
> >>>
> >>> My recently proposed patch adds the first of those to gcc_update, the
> >>> other should be done too.
> >>> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html <https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html>
> >>>
> >>
> >> This script touches files such that they appear more recent than their
> >> dependencies,
> >> so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
> >> For auto* files, this is "fine" as we can run autoreconf or
> >> autoregen.py before starting configure+build, but what about other
> >> files?
> >> For instance, if we have to test a patch which implies changes to
> >> fixincludes/fixincl.x, how should we proceed?
> >> 1- git checkout (with possibly "wrong" timestamps)
> >> 2- apply patch-to-test
> >> 3- contrib/gcc_update -t
> >> 4- configure --enable-maintainer-mode
> >
> > If you ran
> >
> > git reset --hard master // restore state to 'master'
> > contrib/gcc_update // pull latest code
> >
> > then anything coming from upstream will be touched automatically.  You really don't want to re-touch the files after patching unless you're sure they've all been patched correctly, it will break if there's anything regenerated that's missing.
> >
> > R.
>
> Alternatively, if you did
>
> git reset --hard master // restore state to 'master'
> contrib/gcc_update // pull latest code
> patch
> contrib/gcc_update -t
>
> and the second command touched files that weren't listed in the patches, then you'd know that there was something missing from the patch set.  You'd probably still need to go back to the pristine trunk version though, to get the state into something that could be maintainer-mode regenerated.
>

We have been using contrib/gcc_update -t for years (even before the
public CI) because we noticed random timestamp problems after 'git
clone/checkout'. Maybe that was caused by our mirror, I don't
remember.  Anyway after a fresh clone/checkout, we had some random
build failures because the build process was failing to update some
files; probably trying to run the host's too recent autoconf...
Actually I'm not sure about the original purpose of gcc_summary, it
dates back to 1999 and the commit message is a bit... short :-)

The "pull latest code" step you mention is part of what I call "git
clone/checkout".

Anyway, this doesn't fix the problem that --enable-maintainer-mode is broken :-)

Thanks,

Christophe

> R.
>
> >
> >>
> >> I guess --enable-maintainer-mode would be largely (if not completely)
> >> disabled by step 3 above?
> >> And we should probably swap steps 2 and 3, so that the effects of
> >> patch-to-test are handled by make?
> >>
> >> Thanks,
> >>
> >> Christophe
> >>
> >>>
> >>>
> >>>>
> >>>> in binutils/bfd:
> >>>> gdb/sim
> >>>> bfd/po/SRC-POTFILES.in
> >>>> bfd/po/BLD-POTFILES.in
> >>>> bfd/bfd-in2.h
> >>>> bfd/libbfd.h
> >>>> bfd/libcoff.h
> >>>> binutils/po/POTFILES.in
> >>>> gas/po/POTFILES.in
> >>>> opcodes/i386*.h
> >>>> gdb/copying.c
> >>>> gdb/data-directory/*.xml
> >>>> gold/po/POTFILES.in
> >>>> gprof/po/POTFILES.in
> >>>> gfprofng/doc/version.texi
> >>>> ld/po/SRC-POTFILES.in
> >>>> ld/po/BLD-POTFILES.in
> >>>> ld: ldgram/ldlex... and all emulation sources
> >>>> libiberty/functions.texi
> >>>> opcodes/po/POTFILES.in
> >>>> opcodes/aarch64-{asm,dis,opc}-2.c
> >>>> opcodes/ia64 msp430 rl78 rx z8k decoders
> >>>>
> >>>> How are these files "normally" expected to be updated? Do people just
> >>>> manually uncomment the corresponding maintainer lines in the Makefiles
> >>>> and update manually?   In particular we hit issues several times with
> >>>> files under opcodes, that we don't regenerate currently. Maybe we
> >>>> could build binutils in maintainer-mode at -j1 but, well....
> >>>>
> >>>> README-maintainer-mode in binutils/gdb only mentions a problem with
> >>>> 'make distclean' and maintainer mode
> >>>> binutils/README-how-to-make-a-release indicates to use
> >>>> --enable-maintainer-mode, and the sample 'make' invocations do not
> >>>> include any -j flag, is that an indication that only -j1 is supposed
> >>>> to work?
> >>>> Similarly, the src-release.sh script does not use -j.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Christophe
> >>>>
> >>>>>
> >>>>> Grüße
> >>>>>  Thomas
> >

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists)]

On 04/03/2024 16:42, Christophe Lyon wrote:
> On Mon, 4 Mar 2024 at 16:41, Richard Earnshaw <Richard.Earnshaw@arm.com> wrote:
>>
>>
>>
>> On 04/03/2024 15:36, Richard Earnshaw (lists) wrote:
>> > On 04/03/2024 14:46, Christophe Lyon via Gcc wrote:
>> >> On Mon, 4 Mar 2024 at 12:25, Jonathan Wakely <jwakely.gcc@gmail.com> wrote:
>> >>>
>> >>> On Mon, 4 Mar 2024 at 10:44, Christophe Lyon via Gcc <gcc@gcc.gnu.org> wrote:
>> >>>>
>> >>>> Hi!
>> >>>>
>> >>>> On Mon, 4 Mar 2024 at 10:36, Thomas Schwinge <tschwinge@baylibre.com> wrote:
>> >>>>>
>> >>>>> Hi!
>> >>>>>
>> >>>>> On 2024-03-04T00:30:05+0000, Sam James <sam@gentoo.org> wrote:
>> >>>>>> Mark Wielaard <mark@klomp.org> writes:
>> >>>>>>> On Fri, Mar 01, 2024 at 05:32:15PM +0100, Christophe Lyon wrote:
>> >>>>>>>> [...], I read
>> >>>>>>>> https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration> <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration>>
>> >>>>>>>> which basically says "run autoreconf in every dir where there is a
>> >>>>>>>> configure script"
>> >>>>>>>> but this is not exactly what autoregen.py is doing. IIRC it is based
>> >>>>>>>> on a script from Martin Liska, do you know/remember why it follows a
>> >>>>>>>> different process?
>> >>>>>>>
>> >>>>>>> CCing Sam and Arsen who helped refine the autoregen.py script, who
>> >>>>>>> might remember more details. We wanted a script that worked for both
>> >>>>>>> gcc and binutils-gdb. And as far as I know autoreconf simply didn't
>> >>>>>>> work in all directories. We also needed to skip some directories that
>> >>>>>>> did contain a configure script, but that were imported (gotools,
>> >>>>>>> readline, minizip).
>> >>>>>>
>> >>>>>> What we really need to do is, for a start, land tschwinge/aoliva's patches [0]
>> >>>>>> for AC_CONFIG_SUBDIRS.
>> >>>>>
>> >>>>> Let me allocate some time this week to get that one completed.
>> >>>>>
>> >>>>>> Right now, the current situation is janky and it's nowhere near
>> >>>>>> idiomatic autotools usage. It is not a comfortable experience
>> >>>>>> interacting with it even as someone who is familiar and happy with using
>> >>>>>> autotools otherwise.
>> >>>>>>
>> >>>>>> I didn't yet play with maintainer-mode myself but I also didn't see much
>> >>>>>> point given I knew of more fundamental problems like this.
>> >>>>>>
>> >>>>>> [0] https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/ <https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/> <https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/ <https://inbox.sourceware.org/gcc-patches/oril72c4yh.fsf@lxoliva.fsfla.org/>>
>> >>>>>
>> >>>>
>> >>>> Thanks for the background. I didn't follow that discussion at that time :-)
>> >>>>
>> >>>> So... I was confused because I noticed many warnings when doing a simple
>> >>>> find . -name configure |while read f; do echo $f;d=$(dirname $f) &&
>> >>>> autoreconf -f $d && echo $d; done
>> >>>> as suggested by https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration> <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration <https://gcc.gnu.org/wiki/Regenerating_GCC_Configuration>>
>> >>>>
>> >>>> Then I tried with autoregen.py, and saw the same.... and now just
>> >>>> checked Sourceware's bot logs and saw the same numerous warnings at
>> >>>> least in GCC (didn't check binutils yet). Looks like this is
>> >>>> "expected" ....
>> >>>>
>> >>>> I started looking at auto-regenerating these files in our CI a couple
>> >>>> of weeks ago, after we received several "complaints" from contributors
>> >>>> saying that our precommit CI was useless / bothering since it didn't
>> >>>> regenerate files, leading to false alarms.
>> >>>> But now I'm wondering how such contributors regenerate the files
>> >>>> impacted by their patches before committing, they probably just
>> >>>> regenerate things in their subdir of interest, not noticing the whole
>> >>>> picture :-(
>> >>>>
>> >>>> As a first step, we can probably use autoregen.py too, and declare
>> >>>> maintainer-mode broken. However, I do notice that besides the rules
>> >>>> about regenerating configure/Makefile.in/..., maintainer-mode is also
>> >>>> used to update some files.
>> >>>> In gcc:
>> >>>> fixincludes: fixincl.x
>> >>>> libffi: doc/version.texi
>> >>>> libgfortran: some stuff :-)
>> >>>> libiberty: functions.texi
>> >>>
>> >>> My recently proposed patch adds the first of those to gcc_update, the
>> >>> other should be done too.
>> >>> https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html <https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html> <https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html <https://gcc.gnu.org/pipermail/gcc-patches/2024-March/647027.html>>
>> >>>
>> >>
>> >> This script touches files such that they appear more recent than their
>> >> dependencies,
>> >> so IIUC even if one uses --enable-maintainer-mode, it will have no effect.
>> >> For auto* files, this is "fine" as we can run autoreconf or
>> >> autoregen.py before starting configure+build, but what about other
>> >> files?
>> >> For instance, if we have to test a patch which implies changes to
>> >> fixincludes/fixincl.x, how should we proceed?
>> >> 1- git checkout (with possibly "wrong" timestamps)
>> >> 2- apply patch-to-test
>> >> 3- contrib/gcc_update -t
>> >> 4- configure --enable-maintainer-mode
>> >
>> > If you ran
>> >
>> > git reset --hard master // restore state to 'master'
>> > contrib/gcc_update // pull latest code
>> >
>> > then anything coming from upstream will be touched automatically.  You really don't want to re-touch the files after patching unless you're sure they've all been patched correctly, it will break if there's anything regenerated that's missing.
>> >
>> > R.
>>
>> Alternatively, if you did
>>
>> git reset --hard master // restore state to 'master'
>> contrib/gcc_update // pull latest code
>> patch
>> contrib/gcc_update -t
>>
>> and the second command touched files that weren't listed in the patches, then you'd know that there was something missing from the patch set.  You'd probably still need to go back to the pristine trunk version though, to get the state into something that could be maintainer-mode regenerated.
>>
> 
> We have been using contrib/gcc_update -t for years (even before the
> public CI) because we noticed random timestamp problems after 'git
> clone/checkout'. Maybe that was caused by our mirror, I don't
> remember.  Anyway after a fresh clone/checkout, we had some random
> build failures because the build process was failing to update some
> files; probably trying to run the host's too recent autoconf...
> Actually I'm not sure about the original purpose of gcc_summary, it
> dates back to 1999 and the commit message is a bit... short :-)

Tools like git (and svn before it) don't try to maintain time-stamps on patches, the tool just modifies the file and the timestamp comes from the time of the modification.  That's fine if there is nothing regenerated within the repository (it's pure original source), but will cause problems if there are generated files as their time stamps aren't necessarily correct.  `gcc_update --touch` addresses that by ensuring all the generated files are retouched when needed.

R.

> 
> The "pull latest code" step you mention is part of what I call "git
> clone/checkout".
> 
> Anyway, this doesn't fix the problem that --enable-maintainer-mode is broken :-)
> 
> Thanks,
> 
> Christophe
> 
>> R.
>>
>> >
>> >>
>> >> I guess --enable-maintainer-mode would be largely (if not completely)
>> >> disabled by step 3 above?
>> >> And we should probably swap steps 2 and 3, so that the effects of
>> >> patch-to-test are handled by make?
>> >>
>> >> Thanks,
>> >>
>> >> Christophe
>> >>
>> >>>
>> >>>
>> >>>>
>> >>>> in binutils/bfd:
>> >>>> gdb/sim
>> >>>> bfd/po/SRC-POTFILES.in
>> >>>> bfd/po/BLD-POTFILES.in
>> >>>> bfd/bfd-in2.h
>> >>>> bfd/libbfd.h
>> >>>> bfd/libcoff.h
>> >>>> binutils/po/POTFILES.in
>> >>>> gas/po/POTFILES.in
>> >>>> opcodes/i386*.h
>> >>>> gdb/copying.c
>> >>>> gdb/data-directory/*.xml
>> >>>> gold/po/POTFILES.in
>> >>>> gprof/po/POTFILES.in
>> >>>> gfprofng/doc/version.texi
>> >>>> ld/po/SRC-POTFILES.in
>> >>>> ld/po/BLD-POTFILES.in
>> >>>> ld: ldgram/ldlex... and all emulation sources
>> >>>> libiberty/functions.texi
>> >>>> opcodes/po/POTFILES.in
>> >>>> opcodes/aarch64-{asm,dis,opc}-2.c
>> >>>> opcodes/ia64 msp430 rl78 rx z8k decoders
>> >>>>
>> >>>> How are these files "normally" expected to be updated? Do people just
>> >>>> manually uncomment the corresponding maintainer lines in the Makefiles
>> >>>> and update manually?   In particular we hit issues several times with
>> >>>> files under opcodes, that we don't regenerate currently. Maybe we
>> >>>> could build binutils in maintainer-mode at -j1 but, well....
>> >>>>
>> >>>> README-maintainer-mode in binutils/gdb only mentions a problem with
>> >>>> 'make distclean' and maintainer mode
>> >>>> binutils/README-how-to-make-a-release indicates to use
>> >>>> --enable-maintainer-mode, and the sample 'make' invocations do not
>> >>>> include any -j flag, is that an indication that only -j1 is supposed
>> >>>> to work?
>> >>>> Similarly, the src-release.sh script does not use -j.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Christophe
>> >>>>
>> >>>>>
>> >>>>> Grüße
>> >>>>>  Thomas
>> >

Re: Help needed with maintainer-mode

[Vladimir Mezentsev]

On 3/4/24 09:38, Richard Earnshaw (lists) wrote:
> Tools like git (and svn before it) don't try to maintain time-stamps on patches, the tool just modifies the file and the timestamp comes from the time of the modification.  That's fine if there is nothing regenerated within the repository (it's pure original source), but will cause problems if there are generated files as their time stamps aren't necessarily correct.  `gcc_update --touch` addresses that by ensuring all the generated files are retouched when needed.

Why do we save generated files in the source tree?
What will be the problem if we remove Makefile.in and configure from 
source tree and will run `autoreconf -i -f` before building ?

Re: Help needed with maintainer-mode

[Jonathan Wakely]

On Mon, 4 Mar 2024 at 19:27, Vladimir Mezentsev
<vladimir.mezentsev@oracle.com> wrote:
>
>
>
> On 3/4/24 09:38, Richard Earnshaw (lists) wrote:
> > Tools like git (and svn before it) don't try to maintain time-stamps on patches, the tool just modifies the file and the timestamp comes from the time of the modification.  That's fine if there is nothing regenerated within the repository (it's pure original source), but will cause problems if there are generated files as their time stamps aren't necessarily correct.  `gcc_update --touch` addresses that by ensuring all the generated files are retouched when needed.
>
> Why do we save generated files in the source tree?
> What will be the problem if we remove Makefile.in and configure from
> source tree and will run `autoreconf -i -f` before building ?

Having the exact correct versions of autoconf and automake increases
the barrier for new contributors to start work. And to regenerate
everything, they also need autogen, mkinfo, etc.

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists)]

On 04/03/2024 20:04, Jonathan Wakely wrote:
> On Mon, 4 Mar 2024 at 19:27, Vladimir Mezentsev
> <vladimir.mezentsev@oracle.com> wrote:
>>
>>
>>
>> On 3/4/24 09:38, Richard Earnshaw (lists) wrote:
>>> Tools like git (and svn before it) don't try to maintain time-stamps on patches, the tool just modifies the file and the timestamp comes from the time of the modification.  That's fine if there is nothing regenerated within the repository (it's pure original source), but will cause problems if there are generated files as their time stamps aren't necessarily correct.  `gcc_update --touch` addresses that by ensuring all the generated files are retouched when needed.
>>
>> Why do we save generated files in the source tree?
>> What will be the problem if we remove Makefile.in and configure from
>> source tree and will run `autoreconf -i -f` before building ?
> 
> Having the exact correct versions of autoconf and automake increases
> the barrier for new contributors to start work. And to regenerate
> everything, they also need autogen, mkinfo, etc.

It's worse than that.  They might need multiple versions of those tools because different subtrees are built with different, subtly incompatible, versions of those tools.

R.

Re: Help needed with maintainer-mode

[Richard Earnshaw]

On 05/03/2024 14:26, Richard Earnshaw (lists) wrote:
> On 04/03/2024 20:04, Jonathan Wakely wrote:
>> On Mon, 4 Mar 2024 at 19:27, Vladimir Mezentsev
>> <vladimir.mezentsev@oracle.com> wrote:
>>>
>>>
>>>
>>> On 3/4/24 09:38, Richard Earnshaw (lists) wrote:
>>>> Tools like git (and svn before it) don't try to maintain time-stamps on patches, the tool just modifies the file and the timestamp comes from the time of the modification.  That's fine if there is nothing regenerated within the repository (it's pure original source), but will cause problems if there are generated files as their time stamps aren't necessarily correct.  `gcc_update --touch` addresses that by ensuring all the generated files are retouched when needed.
>>>
>>> Why do we save generated files in the source tree?
>>> What will be the problem if we remove Makefile.in and configure from
>>> source tree and will run `autoreconf -i -f` before building ?
>> 
>> Having the exact correct versions of autoconf and automake increases
>> the barrier for new contributors to start work. And to regenerate
>> everything, they also need autogen, mkinfo, etc.
> 
> It's worse than that.  They might need multiple versions of those tools because different subtrees are built with different, subtly incompatible, versions of those tools.
> 
> R.
> 

And I've just remembered another reason as well, which is that some people want to store their sources in a read-only environment; having the tools write to the source area during a build can cause problems (eg if building multiple configurations of the compiler in parallel).

R.

Re: Help needed with maintainer-mode

[Andrew Carlotti]

On Thu, Feb 29, 2024 at 06:39:54PM +0100, Christophe Lyon via Gcc wrote:
> On Thu, 29 Feb 2024 at 12:00, Mark Wielaard <mark@klomp.org> wrote:
> >
> > Hi Christophe,
> >
> > On Thu, Feb 29, 2024 at 11:22:33AM +0100, Christophe Lyon via Gcc wrote:
> > > I've noticed that sourceware's buildbot has a small script
> > > "autoregen.py" which does not use the project's build system, but
> > > rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
> > > Should we replicate that?
> >
> > That python script works across gcc/binutils/gdb:
> > https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
> >
> > It is installed into a container file that has the exact autoconf and
> > automake version needed to regenerate the autotool files:
> > https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
> >
> > And it was indeed done this way because that way the files are
> > regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).
> 
> I see. So it is possibly incomplete, in the sense that it may lack
> some of the steps that maintainer-mode would perform?
> For instance, gas for aarch64 has some *opcodes*.c files that need
> regenerating before committing. The regeneration step is enabled in
> maintainer-mode, so I guess the autoregen bots on Sourceware would
> miss a problem with these files?
> 
> Thanks,
> 
> Christophe

Speaking of opcodes/aarch64-{asm|dis|opc}-2.c - why are these in the source
directory in the first place?  For a similar situation in GCC (gimple-match,
generic-match, insn-emit, etc.) we write the generated files to the build
directory, and generation is always enabled.  I see no reason not to do the
same thing for aarch64-{asm|dis|opc}-2.c.

Andrew

> >
> > It is run on all commits and warns if it detects a change in the
> > (checked in) generated files.
> > https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
> > https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
> >
> > Cheers,
> >
> > Mark

Re: Help needed with maintainer-mode

[Richard Earnshaw (lists)]

On 06/03/2024 15:04, Andrew Carlotti via Gcc wrote:
> On Thu, Feb 29, 2024 at 06:39:54PM +0100, Christophe Lyon via Gcc wrote:
>> On Thu, 29 Feb 2024 at 12:00, Mark Wielaard <mark@klomp.org> wrote:
>>>
>>> Hi Christophe,
>>>
>>> On Thu, Feb 29, 2024 at 11:22:33AM +0100, Christophe Lyon via Gcc wrote:
>>>> I've noticed that sourceware's buildbot has a small script
>>>> "autoregen.py" which does not use the project's build system, but
>>>> rather calls aclocal/autoheader/automake/autoconf in an ad-hoc way.
>>>> Should we replicate that?
>>>
>>> That python script works across gcc/binutils/gdb:
>>> https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
>>>
>>> It is installed into a container file that has the exact autoconf and
>>> automake version needed to regenerate the autotool files:
>>> https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
>>>
>>> And it was indeed done this way because that way the files are
>>> regenerated in a reproducible way. Which wasn't the case when using --enable-maintainer-mode (and autoreconfig also doesn't work).
>>
>> I see. So it is possibly incomplete, in the sense that it may lack
>> some of the steps that maintainer-mode would perform?
>> For instance, gas for aarch64 has some *opcodes*.c files that need
>> regenerating before committing. The regeneration step is enabled in
>> maintainer-mode, so I guess the autoregen bots on Sourceware would
>> miss a problem with these files?
>>
>> Thanks,
>>
>> Christophe
> 
> Speaking of opcodes/aarch64-{asm|dis|opc}-2.c - why are these in the source
> directory in the first place?  For a similar situation in GCC (gimple-match,
> generic-match, insn-emit, etc.) we write the generated files to the build
> directory, and generation is always enabled.  I see no reason not to do the
> same thing for aarch64-{asm|dis|opc}-2.c.
> 

GCC supports building a canadian cross, but binutils doesn't.  To put them in the build area would require setting up host compiler as well and I don't think there's currently enough appetite for doing that extra work.

But every do has its day; maybe the time has come...

R.

> Andrew
> 
>>>
>>> It is run on all commits and warns if it detects a change in the
>>> (checked in) generated files.
>>> https://builder.sourceware.org/buildbot/#/builders/gcc-autoregen
>>> https://builder.sourceware.org/buildbot/#/builders/binutils-gdb-autoregen
>>>
>>> Cheers,
>>>
>>> Mark

Re: Help needed with maintainer-mode

[Thiago Jung Bauermann]

Hello,

Christophe Lyon <christophe.lyon@linaro.org> writes:

> I hoped improving this would be as simple as adding
> --enable-maintainer-mode when configuring, after making sure
> autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
> libtool and gettext seems OK).
>
> However, doing so triggered several problems, which look like race
> conditions in the build system (we build at -j160):
> - random build errors in binutils / gdb with messages like "No rule to
> make target 'po/BLD-POTFILES.in". I managed to reproduce something
> similar manually once, I noticed an empty Makefile; the build logs are
> of course difficult to read, so I couldn't figure out yet what could
> cause this.
>
> - random build failures in gcc in fixincludes. I think this is a race
> condition because fixincludes is updated concurrently both from
> /fixincludes and $buillddir/fixincludes. Probably fixable in gcc
> Makefiles.
>
> - I've seen other errors when building gcc like
> configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
> from libquadmath. I haven't investigated this yet.

I don't know about the last one, but regarding the race conditions, one
workaround might be to define a make target that regenerates all files
(if one doesn't exist already, I don't know) and make the CI call it
with -j1 to avoid concurrency, and then do the regular build step with
-j160.

-- 
Thiago

Re: Help needed with maintainer-mode

[Christophe Lyon]

On Thu, 29 Feb 2024 at 20:49, Thiago Jung Bauermann
<thiago.bauermann@linaro.org> wrote:
>
>
> Hello,
>
> Christophe Lyon <christophe.lyon@linaro.org> writes:
>
> > I hoped improving this would be as simple as adding
> > --enable-maintainer-mode when configuring, after making sure
> > autoconf-2.69 and automake-1.15.1 were in the PATH (using our host's
> > libtool and gettext seems OK).
> >
> > However, doing so triggered several problems, which look like race
> > conditions in the build system (we build at -j160):
> > - random build errors in binutils / gdb with messages like "No rule to
> > make target 'po/BLD-POTFILES.in". I managed to reproduce something
> > similar manually once, I noticed an empty Makefile; the build logs are
> > of course difficult to read, so I couldn't figure out yet what could
> > cause this.
> >
> > - random build failures in gcc in fixincludes. I think this is a race
> > condition because fixincludes is updated concurrently both from
> > /fixincludes and $buillddir/fixincludes. Probably fixable in gcc
> > Makefiles.
> >
> > - I've seen other errors when building gcc like
> > configure.ac:25: error: possibly undefined macro: AM_ENABLE_MULTILIB
> > from libquadmath. I haven't investigated this yet.
>
> I don't know about the last one, but regarding the race conditions, one
> workaround might be to define a make target that regenerates all files
> (if one doesn't exist already, I don't know) and make the CI call it
> with -j1 to avoid concurrency, and then do the regular build step with
> -j160.
>

Yes, that's what I meant below with "magic" make target ;-)

Thanks,

Christophe

> --
> Thiago