[PATCH] Add gnupoke snapshot scheduler, factory and builder

[PATCH] Add gnupoke snapshot scheduler, factory and builder

[Mark Wielaard]

---
 .../containers/Containerfile-fedora-latest    |  1 +
 builder/master.cfg                            | 73 ++++++++++++++++++-
 2 files changed, 70 insertions(+), 4 deletions(-)

diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index e4016af..4fa217b 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -13,6 +13,7 @@ RUN dnf upgrade -y && \
         jq json-c-devel socat \
         texinfo elfutils-debuginfod elfutils-debuginfod-client-devel \
         gcc-plugin-devel binutils-devel \
+        libnbd-devel gc-devel readline-devel texinfo-tex \
         xmlto xmlto-tex fop poppler-utils libxslt docbook-dtds docbook-style-xsl \
         libxml2-devel dos2unix dpkg koji python3-devel python3-koji python3-mock \
         python3-rpm python3-pyxdg mailcap openmpi-devel gdb help2man wget xxhash-devel \
diff --git a/builder/master.cfg b/builder/master.cfg
index 7564f58..fecdc76 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -349,6 +349,15 @@ def master_and_users_try_branches(ref):
     return True
   return False
 
+# For git maint branches (as gnu poke uses)
+def master_and_maint_branches(ref):
+  if ref == "refs/heads/master":
+    return True
+  match = re.match ("refs/heads/maint/poke-[a-zA-Z0-9_-]+", ref)
+  if match:
+    return True
+  return False
+
 sourceware_repourl='https://sourceware.org/git/builder.git'
 sourceware_gitpoller = changes.GitPoller(repourl=sourceware_repourl,
                                          branches=['main'],
@@ -452,7 +461,7 @@ c['change_source'].append(glibc_gitpoller)
 
 gnupoke_repourl='https://git.savannah.gnu.org/git/poke.git'
 gnupoke_gitpoller = changes.GitPoller(repourl=gnupoke_repourl,
-                                      branches=['master'],
+                                      branches=master_and_maint_branches,
                                       pollInterval=3*60,
                                       pollRandomDelayMax=2*60,
                                       project='gnupoke')
@@ -982,6 +991,16 @@ gnupoke_scheduler = schedulers.SingleBranchScheduler(
                       "gnupoke-gentoo-sparc"])
 c['schedulers'].append(gnupoke_scheduler)
 
+gnupoke_snapshot_scheduler = schedulers.Periodic(
+        name="gnupoke-snapshots",
+        change_filter=util.ChangeFilter(project="gnupoke",
+                                        branch_re=re.compile("master$|poke-[a-zA-Z0-9_-]+$")),
+        periodicBuildTimer=15*60, # 15 minutes in seconds
+        onlyIfChanged=True,
+        reason="gnupoke periodic project master/maint branch snapshot",
+        builderNames=["poke-snapshots-html-pdf-dist"])
+c['schedulers'].append(gnupoke_snapshot_scheduler)
+
 annobin_scheduler = schedulers.SingleBranchScheduler(
         name="annobin",
         change_filter=util.ChangeFilter(project="annobin",
@@ -1082,6 +1101,12 @@ make_step = steps.Compile(
             util.Interpolate('-j%(prop:ncpus)s')]),
         name="make",
         haltOnFailure=True)
+make_html_step = steps.Compile(
+        command=["make", "html"],
+        name="make html")
+make_pdf_step = steps.Compile(
+        command=["make", "pdf"],
+        name="make pdf")
 make_check_step = steps.Test(
         command=addOutputSync.withArgs(["make", "check",
             util.Interpolate('-j%(prop:ncpus)s')]),
@@ -1749,6 +1774,11 @@ elfutils_create_publish_file_step = steps.ShellCommand(
         name="create publish file",
         command="echo elfutils/coverage > /home/builder/shared/publish")
 
+# publish file based on branch name, cut away all but last / element
+poke_create_publish_file_step = steps.ShellCommand(
+        name="create publish file",
+        command=util.Interpolate('(echo -n gnupoke/; echo %(prop:branch)s | rev | cut -f1 -d/ | rev | sed -e s/master/trunk/) > /home/builder/shared/publish'))
+
 # Upload automake test results, autoconf config.log, in both build and dist-build trees
 elfutils_upload_to_bunsen = bunsen_logfile_upload_cpio_steps(
         ["*.trs", "*.log"])
@@ -4275,13 +4305,25 @@ systemtap_gentoo_sparc_builder = util.BuilderConfig(
 c['builders'].append(systemtap_gentoo_sparc_builder)
 
 # GNU poke builders and factories
-gnupoke_factory = util.BuildFactory()
-gnupoke_factory.addStep(steps.Git(
+
+# publish file based on branch name, cut away all but last / element
+poke_create_publish_file_step = steps.ShellCommand(
+        name="create publish file",
+        command=util.Interpolate('(echo -n gnupoke/; echo %(prop:branch)s | rev | cut -f1 -d/ | rev | sed -e s/master/trunk/) > /home/builder/shared/publish'))
+
+poke_create_output_step = steps.ShellCommand(
+        name="create output",
+        command="mkdir /home/builder/shared/output && mv ./poke-*.tar.gz /home/builder/shared/output/ && mv ./doc/poke.html /home/builder/shared/output/html && mv ./doc/poke.pdf /home/builder/shared/output/")
+
+poke_git_step = steps.Git(
         repourl=gnupoke_repourl,
         mode='full', method='fresh',
         name="git checkout",
         getDescription={'tags':True, 'always':True},        
-        haltOnFailure=True))
+        haltOnFailure=True)
+
+gnupoke_factory = util.BuildFactory()
+gnupoke_factory.addStep(poke_git_step)
 gnupoke_factory.addStep(bootstrap_step)
 gnupoke_factory.addStep(configure_step)
 gnupoke_factory.addStep(make_step)
@@ -4291,6 +4333,19 @@ gnupoke_factory.addStep(steps.Test(command=["make", "syntax-check"],
                                    haltOnFailure=False, flunkOnFailure=True))
 gnupoke_factory.addSteps(bunsen_logfile_upload_cpio_steps(["*.sum", "*.log"]))
 
+# snapshots
+gnupoke_snapshots_factory = util.BuildFactory()
+gnupoke_snapshots_factory.addStep(poke_git_step)
+gnupoke_snapshots_factory.addStep(bootstrap_step)
+gnupoke_snapshots_factory.addStep(configure_step)
+gnupoke_snapshots_factory.addStep(make_step)
+gnupoke_snapshots_factory.addStep(make_html_step)
+gnupoke_snapshots_factory.addStep(make_pdf_step)
+gnupoke_snapshots_factory.addStep(make_dist_step)
+gnupoke_snapshots_factory.addStep(wait_snapshots_output_ready_step)
+gnupoke_snapshots_factory.addStep(poke_create_output_step)
+gnupoke_snapshots_factory.addStep(poke_create_publish_file_step)
+
 gnupoke_debiantesting_x86_64_builder = util.BuilderConfig(
         name="gnupoke-debian-testing-x86_64",
         tags=["gnupoke", "debian-testing", "x86_64"],
@@ -4314,6 +4369,16 @@ gnupoke_gentoo_space_builder = util.BuilderConfig(
         factory=gnupoke_factory)
 c['builders'].append(gnupoke_gentoo_space_builder)
 
+poke_snapshots_builder = util.BuilderConfig(
+        name="poke-snapshots-html-pdf-dist",
+        collapseRequests=True,
+        properties={'container-file':
+                    readContainerFile('fedora-latest')},
+        workernames="snapshots",
+        tags=["gnupoke-snapshots"],
+        factory=gnupoke_snapshots_factory)
+c['builders'].append(poke_snapshots_builder)
+
 # annobin steps, builders and factories
 
 # annobin checkouts have generated config files included, but might
-- 
2.31.1

Re: [PATCH] Add gnupoke snapshot scheduler, factory and builder

[Jose E. Marchesi]

Thanks Mark! :)

This and the Fedora ARM support means I should add new little graphics
to http://jemarch.net/poke?

> ---
>  .../containers/Containerfile-fedora-latest    |  1 +
>  builder/master.cfg                            | 73 ++++++++++++++++++-
>  2 files changed, 70 insertions(+), 4 deletions(-)
>
> diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
> index e4016af..4fa217b 100644
> --- a/builder/containers/Containerfile-fedora-latest
> +++ b/builder/containers/Containerfile-fedora-latest
> @@ -13,6 +13,7 @@ RUN dnf upgrade -y && \
>          jq json-c-devel socat \
>          texinfo elfutils-debuginfod elfutils-debuginfod-client-devel \
>          gcc-plugin-devel binutils-devel \
> +        libnbd-devel gc-devel readline-devel texinfo-tex \
>          xmlto xmlto-tex fop poppler-utils libxslt docbook-dtds docbook-style-xsl \
>          libxml2-devel dos2unix dpkg koji python3-devel python3-koji python3-mock \
>          python3-rpm python3-pyxdg mailcap openmpi-devel gdb help2man wget xxhash-devel \
> diff --git a/builder/master.cfg b/builder/master.cfg
> index 7564f58..fecdc76 100644
> --- a/builder/master.cfg
> +++ b/builder/master.cfg
> @@ -349,6 +349,15 @@ def master_and_users_try_branches(ref):
>      return True
>    return False
>  
> +# For git maint branches (as gnu poke uses)
> +def master_and_maint_branches(ref):
> +  if ref == "refs/heads/master":
> +    return True
> +  match = re.match ("refs/heads/maint/poke-[a-zA-Z0-9_-]+", ref)
> +  if match:
> +    return True
> +  return False
> +
>  sourceware_repourl='https://sourceware.org/git/builder.git'
>  sourceware_gitpoller = changes.GitPoller(repourl=sourceware_repourl,
>                                           branches=['main'],
> @@ -452,7 +461,7 @@ c['change_source'].append(glibc_gitpoller)
>  
>  gnupoke_repourl='https://git.savannah.gnu.org/git/poke.git'
>  gnupoke_gitpoller = changes.GitPoller(repourl=gnupoke_repourl,
> -                                      branches=['master'],
> +                                      branches=master_and_maint_branches,
>                                        pollInterval=3*60,
>                                        pollRandomDelayMax=2*60,
>                                        project='gnupoke')
> @@ -982,6 +991,16 @@ gnupoke_scheduler = schedulers.SingleBranchScheduler(
>                        "gnupoke-gentoo-sparc"])
>  c['schedulers'].append(gnupoke_scheduler)
>  
> +gnupoke_snapshot_scheduler = schedulers.Periodic(
> +        name="gnupoke-snapshots",
> +        change_filter=util.ChangeFilter(project="gnupoke",
> +                                        branch_re=re.compile("master$|poke-[a-zA-Z0-9_-]+$")),
> +        periodicBuildTimer=15*60, # 15 minutes in seconds
> +        onlyIfChanged=True,
> +        reason="gnupoke periodic project master/maint branch snapshot",
> +        builderNames=["poke-snapshots-html-pdf-dist"])
> +c['schedulers'].append(gnupoke_snapshot_scheduler)
> +
>  annobin_scheduler = schedulers.SingleBranchScheduler(
>          name="annobin",
>          change_filter=util.ChangeFilter(project="annobin",
> @@ -1082,6 +1101,12 @@ make_step = steps.Compile(
>              util.Interpolate('-j%(prop:ncpus)s')]),
>          name="make",
>          haltOnFailure=True)
> +make_html_step = steps.Compile(
> +        command=["make", "html"],
> +        name="make html")
> +make_pdf_step = steps.Compile(
> +        command=["make", "pdf"],
> +        name="make pdf")
>  make_check_step = steps.Test(
>          command=addOutputSync.withArgs(["make", "check",
>              util.Interpolate('-j%(prop:ncpus)s')]),
> @@ -1749,6 +1774,11 @@ elfutils_create_publish_file_step = steps.ShellCommand(
>          name="create publish file",
>          command="echo elfutils/coverage > /home/builder/shared/publish")
>  
> +# publish file based on branch name, cut away all but last / element
> +poke_create_publish_file_step = steps.ShellCommand(
> +        name="create publish file",
> +        command=util.Interpolate('(echo -n gnupoke/; echo %(prop:branch)s | rev | cut -f1 -d/ | rev | sed -e s/master/trunk/) > /home/builder/shared/publish'))
> +
>  # Upload automake test results, autoconf config.log, in both build and dist-build trees
>  elfutils_upload_to_bunsen = bunsen_logfile_upload_cpio_steps(
>          ["*.trs", "*.log"])
> @@ -4275,13 +4305,25 @@ systemtap_gentoo_sparc_builder = util.BuilderConfig(
>  c['builders'].append(systemtap_gentoo_sparc_builder)
>  
>  # GNU poke builders and factories
> -gnupoke_factory = util.BuildFactory()
> -gnupoke_factory.addStep(steps.Git(
> +
> +# publish file based on branch name, cut away all but last / element
> +poke_create_publish_file_step = steps.ShellCommand(
> +        name="create publish file",
> +        command=util.Interpolate('(echo -n gnupoke/; echo %(prop:branch)s | rev | cut -f1 -d/ | rev | sed -e s/master/trunk/) > /home/builder/shared/publish'))
> +
> +poke_create_output_step = steps.ShellCommand(
> +        name="create output",
> +        command="mkdir /home/builder/shared/output && mv ./poke-*.tar.gz /home/builder/shared/output/ && mv ./doc/poke.html /home/builder/shared/output/html && mv ./doc/poke.pdf /home/builder/shared/output/")
> +
> +poke_git_step = steps.Git(
>          repourl=gnupoke_repourl,
>          mode='full', method='fresh',
>          name="git checkout",
>          getDescription={'tags':True, 'always':True},        
> -        haltOnFailure=True))
> +        haltOnFailure=True)
> +
> +gnupoke_factory = util.BuildFactory()
> +gnupoke_factory.addStep(poke_git_step)
>  gnupoke_factory.addStep(bootstrap_step)
>  gnupoke_factory.addStep(configure_step)
>  gnupoke_factory.addStep(make_step)
> @@ -4291,6 +4333,19 @@ gnupoke_factory.addStep(steps.Test(command=["make", "syntax-check"],
>                                     haltOnFailure=False, flunkOnFailure=True))
>  gnupoke_factory.addSteps(bunsen_logfile_upload_cpio_steps(["*.sum", "*.log"]))
>  
> +# snapshots
> +gnupoke_snapshots_factory = util.BuildFactory()
> +gnupoke_snapshots_factory.addStep(poke_git_step)
> +gnupoke_snapshots_factory.addStep(bootstrap_step)
> +gnupoke_snapshots_factory.addStep(configure_step)
> +gnupoke_snapshots_factory.addStep(make_step)
> +gnupoke_snapshots_factory.addStep(make_html_step)
> +gnupoke_snapshots_factory.addStep(make_pdf_step)
> +gnupoke_snapshots_factory.addStep(make_dist_step)
> +gnupoke_snapshots_factory.addStep(wait_snapshots_output_ready_step)
> +gnupoke_snapshots_factory.addStep(poke_create_output_step)
> +gnupoke_snapshots_factory.addStep(poke_create_publish_file_step)
> +
>  gnupoke_debiantesting_x86_64_builder = util.BuilderConfig(
>          name="gnupoke-debian-testing-x86_64",
>          tags=["gnupoke", "debian-testing", "x86_64"],
> @@ -4314,6 +4369,16 @@ gnupoke_gentoo_space_builder = util.BuilderConfig(
>          factory=gnupoke_factory)
>  c['builders'].append(gnupoke_gentoo_space_builder)
>  
> +poke_snapshots_builder = util.BuilderConfig(
> +        name="poke-snapshots-html-pdf-dist",
> +        collapseRequests=True,
> +        properties={'container-file':
> +                    readContainerFile('fedora-latest')},
> +        workernames="snapshots",
> +        tags=["gnupoke-snapshots"],
> +        factory=gnupoke_snapshots_factory)
> +c['builders'].append(poke_snapshots_builder)
> +
>  # annobin steps, builders and factories
>  
>  # annobin checkouts have generated config files included, but might

Re: [PATCH] Add gnupoke snapshot scheduler, factory and builder

[Mark Wielaard]

Hi Jose,

On Sun, Apr 16, 2023 at 09:53:34PM +0200, Jose E. Marchesi wrote:
> This and the Fedora ARM support means I should add new little graphics
> to http://jemarch.net/poke?

For the fedora arm64 builder you could add:

<a href="https://builder.sourceware.org/buildbot/#builders/gnupoke-fedora-arm64"><img src="https://builder.sourceware.org/buildbot/badges/gnupoke-fedora-arm64.svg?left_text=%20%20fedora-arm64%20%20"></a>

For the snapshots we should see where they exactly land when there is
a real build. It was supposed to go into "trunk" or "poke-[123]" under
https://snapshots.sourceware.org/gnupoke/ but the first build didn't
have a branch associated with it, so it moved a level too high.

Please push a new commit to top of git or one of the maint/poke-*
branches, so we can check where they end up exactly.

Cheers,

Mark

Re: [PATCH] Add gnupoke snapshot scheduler, factory and builder

[Jose E. Marchesi]

Hi Mark.

> On Sun, Apr 16, 2023 at 09:53:34PM +0200, Jose E. Marchesi wrote:
>> This and the Fedora ARM support means I should add new little graphics
>> to http://jemarch.net/poke?
>
> For the fedora arm64 builder you could add:
>
> <a
> href="https://builder.sourceware.org/buildbot/#builders/gnupoke-fedora-arm64"><img
> src="https://builder.sourceware.org/buildbot/badges/gnupoke-fedora-arm64.svg?left_text=%20%20fedora-arm64%20%20"></a>

I just added the above in https://jemarch.net/poke.

> For the snapshots we should see where they exactly land when there is
> a real build. It was supposed to go into "trunk" or "poke-[123]" under
> https://snapshots.sourceware.org/gnupoke/ but the first build didn't
> have a branch associated with it, so it moved a level too high.
>
> Please push a new commit to top of git or one of the maint/poke-*
> branches, so we can check where they end up exactly.

I just pushed a commit to `master'.

Re: [PATCH] Add gnupoke snapshot scheduler, factory and builder

[Mark Wielaard]

On Mon, 2023-04-17 at 11:29 +0200, Jose E. Marchesi wrote:
> Hi Mark.
> 
> > On Sun, Apr 16, 2023 at 09:53:34PM +0200, Jose E. Marchesi wrote:
> > > This and the Fedora ARM support means I should add new little graphics
> > > to http://jemarch.net/poke?
> > 
> > For the fedora arm64 builder you could add:
> > 
> > <a
> > href="https://builder.sourceware.org/buildbot/#builders/gnupoke-fedora-arm64"><img
> > src="https://builder.sourceware.org/buildbot/badges/gnupoke-fedora-arm64.svg?left_text=%20%20fedora-arm64%20%20"></a>
> 
> I just added the above in https://jemarch.net/poke.

And they are all green now \o/

> > For the snapshots we should see where they exactly land when there is
> > a real build. It was supposed to go into "trunk" or "poke-[123]" under
> > https://snapshots.sourceware.org/gnupoke/ but the first build didn't
> > have a branch associated with it, so it moved a level too high.
> > 
> > Please push a new commit to top of git or one of the maint/poke-*
> > branches, so we can check where they end up exactly.
> 
> I just pushed a commit to `master'.

Great, and that worked as expected, see
https://snapshots.sourceware.org/gnupoke/trunk/latest/

Cheers,

Mark