Re: Checks that autotools generated files were re-generated correctly

Re: Checks that autotools generated files were re-generated correctly

[Mark Wielaard]

Hi! (adding gdb and binutils to the CC)

On Thu, Nov 09, 2023 at 12:30:59AM +0100, Mark Wielaard wrote:
> On Mon, Nov 06, 2023 at 06:04:48PM +0100, Martin Jambor wrote:
> > I have inherited Martin Liška's buildbot script that checks that all
> > sorts of autotools generated files, mainly configure scripts, were
> > re-generated correctly when appropriate.  While the checks are hopefully
> > useful, they report issues surprisingly often and reporting them feels
> > especially unproductive.
> 
> Cool! A small python script cannot replace him of course. But it is
> nice to get a small virtual mliska :)
> 
> > Could such checks be added to our server side push hooks so that commits
> > introducing these breakages would get refused automatically.  While the
> > check might be a bit expensive, it only needs to be run on files
> > touching the generated files and/or the files these are generated from.
> 
> So this doesn't just need that script, but also an execution
> environment that contains the right versions of the autotools. We
> could install those of course, but running them from a git hook on
> checkin indeed seems a little expensive.
> 
> Creating a container with the script and the right versions of all
> tools might be the best thing. Then the script can be run from a cron
> job or buildbot. Or even by someone hacking on the build/configure
> scripts to make sure they are generating with the right tools.
> 
> builder.sourceware.org already contains various of such containers:
> https://sourceware.org/cgit/builder/tree/builder/containers
> https://sourceware.org/cgit/builder/tree/README_containers
> 
> Friday is Sourceware Open Office hour (#overseers on irc.libera.chat
> at 18:00 UTC). We could hack something together then and see how to
> hook it up.

So we did indeed discuss and hack something together.

The container file is here:
https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools

The script is here:
https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py

A buildbot can then use that container to run that script and then
check that git diff is empty on every commit to binutils-gdb.git and
gcc.git. If it finds an issue it will sent email with the diff.

It already found issues in both gcc and binutils-gdb. All fixed now.

Thanks to Arsen and Sam for testing, fixing and updating the script
(it now also runs aclocal).

If you want to run this locally you can use the container file to
create an image and then run autoregen.py on your local tree by bind
mounting your git tree inside it.

$ git clone https://sourceware.org/git/builder.git
$ cd builder/
$ podman build -t autotools -f builder/containers/Containerfile-autotools \
  builder/containers
$ cd .. # assuming your binutils-gdb directory is here
$ podman run --privileged -u root -v $(pwd)/binutils-gdb:/binutils-gdb \
  -ti --entrypoint "/bin/bash" autotools
# then inside the container
 cd /binutils-gdb
 autoregen.py

Change binutils-gdb to gcc if you are working on gcc.
You should also be able to do the above with docker instead of podman.

Cheers,

Mark

Re: Checks that autotools generated files were re-generated correctly

[Martin Jambor]

Hi,

On Wed, Nov 15 2023, Mark Wielaard wrote:
> Hi! (adding gdb and binutils to the CC)
>
> On Thu, Nov 09, 2023 at 12:30:59AM +0100, Mark Wielaard wrote:
>> On Mon, Nov 06, 2023 at 06:04:48PM +0100, Martin Jambor wrote:
>> > I have inherited Martin Liška's buildbot script that checks that all
>> > sorts of autotools generated files, mainly configure scripts, were
>> > re-generated correctly when appropriate.  While the checks are hopefully
>> > useful, they report issues surprisingly often and reporting them feels
>> > especially unproductive.
>> 
>> Cool! A small python script cannot replace him of course. But it is
>> nice to get a small virtual mliska :)
>> 
>> > Could such checks be added to our server side push hooks so that commits
>> > introducing these breakages would get refused automatically.  While the
>> > check might be a bit expensive, it only needs to be run on files
>> > touching the generated files and/or the files these are generated from.
>> 
>> So this doesn't just need that script, but also an execution
>> environment that contains the right versions of the autotools. We
>> could install those of course, but running them from a git hook on
>> checkin indeed seems a little expensive.
>> 
>> Creating a container with the script and the right versions of all
>> tools might be the best thing. Then the script can be run from a cron
>> job or buildbot. Or even by someone hacking on the build/configure
>> scripts to make sure they are generating with the right tools.
>> 
>> builder.sourceware.org already contains various of such containers:
>> https://sourceware.org/cgit/builder/tree/builder/containers
>> https://sourceware.org/cgit/builder/tree/README_containers
>> 
>> Friday is Sourceware Open Office hour (#overseers on irc.libera.chat
>> at 18:00 UTC). We could hack something together then and see how to
>> hook it up.
>
> So we did indeed discuss and hack something together.

Wonderful, thanks to everyone working to improve the situation with
these errors.

Martin

>
> The container file is here:
> https://sourceware.org/cgit/builder/tree/builder/containers/Containerfile-autotools
>
> The script is here:
> https://sourceware.org/cgit/builder/tree/builder/containers/autoregen.py
>
> A buildbot can then use that container to run that script and then
> check that git diff is empty on every commit to binutils-gdb.git and
> gcc.git. If it finds an issue it will sent email with the diff.
>
> It already found issues in both gcc and binutils-gdb. All fixed now.
>
> Thanks to Arsen and Sam for testing, fixing and updating the script
> (it now also runs aclocal).
>
> If you want to run this locally you can use the container file to
> create an image and then run autoregen.py on your local tree by bind
> mounting your git tree inside it.
>
> $ git clone https://sourceware.org/git/builder.git
> $ cd builder/
> $ podman build -t autotools -f builder/containers/Containerfile-autotools \
>   builder/containers
> $ cd .. # assuming your binutils-gdb directory is here
> $ podman run --privileged -u root -v $(pwd)/binutils-gdb:/binutils-gdb \
>   -ti --entrypoint "/bin/bash" autotools
> # then inside the container
>  cd /binutils-gdb
>  autoregen.py
>
> Change binutils-gdb to gcc if you are working on gcc.
> You should also be able to do the above with docker instead of podman.
>
> Cheers,
>
> Mark