[PATCH] Add binutils-gdb-autoregen builder

[PATCH] Add binutils-gdb-autoregen builder

[Mark Wielaard]

Generalize the autoregen.py script by searching for 'configure.ac'
instead of 'configure'. Skip "foreign" readline and minizip dirs.

Make autoregen_step and git_diff_step generic, define early.
Add binutils-gdb-autoregen builder to binutils_gdb_scheduler.
---
 builder/containers/autoregen.py |  6 ++++-
 builder/master.cfg              | 47 ++++++++++++++++++++++++---------
 2 files changed, 39 insertions(+), 14 deletions(-)

diff --git a/builder/containers/autoregen.py b/builder/containers/autoregen.py
index 4ebf445..9315334 100755
--- a/builder/containers/autoregen.py
+++ b/builder/containers/autoregen.py
@@ -15,11 +15,15 @@ config_folders = []
 
 for root, _, files in os.walk('.'):
     for file in files:
-        if file == 'configure':
+        if file == 'configure.ac':
             config_folders.append(Path(root).resolve())
 
 for folder in sorted(config_folders):
     print(folder, flush=True)
+    # readline and minizip are maintained with different autotools versions
+    if (str(folder).endswith('readline')
+        or str(folder).endswith('minizip')):
+        continue;
     os.chdir(folder)
     configure_lines = open('configure.ac').read().splitlines()
     if any(True for line in configure_lines if line.startswith('AC_CONFIG_HEADERS')):
diff --git a/builder/master.cfg b/builder/master.cfg
index 0fef831..51c12b9 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -906,6 +906,7 @@ binutils_gdb_scheduler = schedulers.SingleBranchScheduler(
                                         branch="master"),
         reason="binutils-gdb project master branch update",
         builderNames=["binutils-gdb-fedrawhide-x86_64",
+            "binutils-gdb-autoregen",
             "binutils-gdb-clang-fedrawhide-x86_64",
             "binutils-gdb-gentoo-sparc"])
 c['schedulers'].append(binutils_gdb_scheduler)
@@ -1182,6 +1183,16 @@ make_distcheck_testsuite_step = steps.Test(
         logfiles={"testsuite.log":
                   util.Interpolate('%(prop:package_name)s-%(prop:package_version)s/_build/sub/tests/testsuite.log')})
 
+autoregen_step = steps.ShellCommand(
+        command="autoregen.py",
+        name="autoregen.py",
+        haltOnFailure=True)
+
+git_diff_step = steps.ShellCommand(
+        command=['git', 'diff', '--exit-code'],
+        name="git diff",
+        haltOnFailure=True)
+
 # Generic make clean step to be run at the end of a build
 make_clean_step = steps.ShellCommand(
         command=["make", "clean"],
@@ -2470,6 +2481,13 @@ binutils_step_git = steps.Git(
         getDescription={'tags':True, 'always':True},
         name="git checkout",
         haltOnFailure=True)
+binutils_gdb_step_git = steps.Git(
+        repourl=binutils_gdb_repourl,
+        mode='full', method='fresh',
+        retryFetch=True, retry=(30,3),
+        getDescription={'tags':True, 'always':True},
+        name="git checkout",
+        haltOnFailure=True)
 binutils_step_rm_build_dir = steps.ShellCommand(
         command=["rm", "-rf",
                  util.Interpolate ("%(prop:builddir)s/binutils-build")],
@@ -2598,6 +2616,11 @@ binutils_factory_gas_binutils.addStep(binutils_step_check_gas_binutils)
 binutils_factory_gas_binutils.addSteps(binutils_steps_bunsen_gas_binutils)
 binutils_factory_gas_binutils.addStep(binutils_step_rm_build_dir)
 
+binutils_gdb_autoregen_factory = util.BuildFactory()
+binutils_gdb_autoregen_factory.addStep(binutils_gdb_step_git)
+binutils_gdb_autoregen_factory.addStep(autoregen_step)
+binutils_gdb_autoregen_factory.addStep(git_diff_step)
+
 binutils_debian_amd64_builder = util.BuilderConfig(
 	name="binutils-debian-amd64",
         properties={'container-file':
@@ -2799,6 +2822,15 @@ binutils_try_debian_testing_x86_64_builder = util.BuilderConfig(
         factory=binutils_factory_libctf)
 c['builders'].append(binutils_try_debian_testing_x86_64_builder)
 
+binutils_gdb_autoregen_builder = util.BuilderConfig(
+        name="binutils-gdb-autoregen",
+        properties={'container-file':
+                    readContainerFile('autotools')},
+        workernames=vm_workers,
+        tags=["binutils-gdb-autotools", "debian"],
+        factory=binutils_gdb_autoregen_factory)
+c['builders'].append(binutils_gdb_autoregen_builder)
+
 # gdb build steps, factory and builders
 
 gdb_git_step = steps.Git(
@@ -3384,16 +3416,6 @@ gcc_build_make_clean_step = steps.ShellCommand(
         command=["make", "clean"],
         name="make clean")
 
-gcc_autoregen_step = steps.ShellCommand(
-        command="autoregen.py",
-        name="autoregen.py",
-        haltOnFailure=True)
-
-gcc_git_diff_step = steps.ShellCommand(
-        command=['git', 'diff', '--exit-code'],
-        name="git diff",
-        haltOnFailure=True)
-
 gcc_factory = util.BuildFactory()
 gcc_factory.addStep(gcc_git_step)
 gcc_factory.addStep(steps.Configure(
@@ -3441,12 +3463,11 @@ gcc_build_factory.addStep(gcc_rm_build_step)
 
 gcc_autoregen_factory = util.BuildFactory()
 gcc_autoregen_factory.addStep(gcc_git_step)
-gcc_autoregen_factory.addStep(gcc_autoregen_step)
-gcc_autoregen_factory.addStep(gcc_git_diff_step)
+gcc_autoregen_factory.addStep(autoregen_step)
+gcc_autoregen_factory.addStep(git_diff_step)
 
 gcc_autoregen_builder = util.BuilderConfig(
         name="gcc-autoregen",
-        collapseRequests=True,
         properties={'container-file':
                     readContainerFile('autotools')},
         workernames=vm_workers,
-- 
2.39.3

Re: [PATCH] Add binutils-gdb-autoregen builder

[Frank Ch. Eigler]

Hi -

> Generalize the autoregen.py script by searching for 'configure.ac'
> instead of 'configure'. Skip "foreign" readline and minizip dirs.
> Make autoregen_step and git_diff_step generic, define early.
> Add binutils-gdb-autoregen builder to binutils_gdb_scheduler.

> +autoregen_step = steps.ShellCommand(
> +        command="autoregen.py",
> +        name="autoregen.py",
> +        haltOnFailure=True)

Note that this stuff literally can only add work for the developers.
If the tooling knows what the output purportedly should be, it is in a
position to correct the mismatches, not merely complain about it.
That would actually save human effort.

- FChE

Re: [PATCH] Add binutils-gdb-autoregen builder

[Mark Wielaard]

Hi Frank,

On Sun, Nov 12, 2023 at 11:49:11AM -0500, Frank Ch. Eigler wrote:
> > Generalize the autoregen.py script by searching for 'configure.ac'
> > instead of 'configure'. Skip "foreign" readline and minizip dirs.
> > Make autoregen_step and git_diff_step generic, define early.
> > Add binutils-gdb-autoregen builder to binutils_gdb_scheduler.
> 
> > +autoregen_step = steps.ShellCommand(
> > +        command="autoregen.py",
> > +        name="autoregen.py",
> > +        haltOnFailure=True)
> 
> Note that this stuff literally can only add work for the developers.
> If the tooling knows what the output purportedly should be, it is in a
> position to correct the mismatches, not merely complain about it.
> That would actually save human effort.

The build step does generate a diff that when applies makes sure the
files are regenerated with the correct version of the tools. The
buildbots however don't have write access, so cannot apply and push
the patch itself.

The container can also be used to generate the same locally, see
https://inbox.sourceware.org/20231112171138.1553448-1-mark@klomp.org/T/

So it could also be done through some other mechanism which has direct
write access to the git repro. But you'll have to negotiate with the
projects to see if they want that.

Cheers,

Mark