* [PATCH] Add gnupoke snapshot scheduler, factory and builder
@ 2023-04-16 19:05 Mark Wielaard
2023-04-16 19:53 ` Jose E. Marchesi
0 siblings, 1 reply; 5+ messages in thread
From: Mark Wielaard @ 2023-04-16 19:05 UTC (permalink / raw)
To: buildbot; +Cc: Jose E. Marchesi, Mark Wielaard
---
.../containers/Containerfile-fedora-latest | 1 +
builder/master.cfg | 73 ++++++++++++++++++-
2 files changed, 70 insertions(+), 4 deletions(-)
diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
index e4016af..4fa217b 100644
--- a/builder/containers/Containerfile-fedora-latest
+++ b/builder/containers/Containerfile-fedora-latest
@@ -13,6 +13,7 @@ RUN dnf upgrade -y && \
jq json-c-devel socat \
texinfo elfutils-debuginfod elfutils-debuginfod-client-devel \
gcc-plugin-devel binutils-devel \
+ libnbd-devel gc-devel readline-devel texinfo-tex \
xmlto xmlto-tex fop poppler-utils libxslt docbook-dtds docbook-style-xsl \
libxml2-devel dos2unix dpkg koji python3-devel python3-koji python3-mock \
python3-rpm python3-pyxdg mailcap openmpi-devel gdb help2man wget xxhash-devel \
diff --git a/builder/master.cfg b/builder/master.cfg
index 7564f58..fecdc76 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -349,6 +349,15 @@ def master_and_users_try_branches(ref):
return True
return False
+# For git maint branches (as gnu poke uses)
+def master_and_maint_branches(ref):
+ if ref == "refs/heads/master":
+ return True
+ match = re.match ("refs/heads/maint/poke-[a-zA-Z0-9_-]+", ref)
+ if match:
+ return True
+ return False
+
sourceware_repourl='https://sourceware.org/git/builder.git'
sourceware_gitpoller = changes.GitPoller(repourl=sourceware_repourl,
branches=['main'],
@@ -452,7 +461,7 @@ c['change_source'].append(glibc_gitpoller)
gnupoke_repourl='https://git.savannah.gnu.org/git/poke.git'
gnupoke_gitpoller = changes.GitPoller(repourl=gnupoke_repourl,
- branches=['master'],
+ branches=master_and_maint_branches,
pollInterval=3*60,
pollRandomDelayMax=2*60,
project='gnupoke')
@@ -982,6 +991,16 @@ gnupoke_scheduler = schedulers.SingleBranchScheduler(
"gnupoke-gentoo-sparc"])
c['schedulers'].append(gnupoke_scheduler)
+gnupoke_snapshot_scheduler = schedulers.Periodic(
+ name="gnupoke-snapshots",
+ change_filter=util.ChangeFilter(project="gnupoke",
+ branch_re=re.compile("master$|poke-[a-zA-Z0-9_-]+$")),
+ periodicBuildTimer=15*60, # 15 minutes in seconds
+ onlyIfChanged=True,
+ reason="gnupoke periodic project master/maint branch snapshot",
+ builderNames=["poke-snapshots-html-pdf-dist"])
+c['schedulers'].append(gnupoke_snapshot_scheduler)
+
annobin_scheduler = schedulers.SingleBranchScheduler(
name="annobin",
change_filter=util.ChangeFilter(project="annobin",
@@ -1082,6 +1101,12 @@ make_step = steps.Compile(
util.Interpolate('-j%(prop:ncpus)s')]),
name="make",
haltOnFailure=True)
+make_html_step = steps.Compile(
+ command=["make", "html"],
+ name="make html")
+make_pdf_step = steps.Compile(
+ command=["make", "pdf"],
+ name="make pdf")
make_check_step = steps.Test(
command=addOutputSync.withArgs(["make", "check",
util.Interpolate('-j%(prop:ncpus)s')]),
@@ -1749,6 +1774,11 @@ elfutils_create_publish_file_step = steps.ShellCommand(
name="create publish file",
command="echo elfutils/coverage > /home/builder/shared/publish")
+# publish file based on branch name, cut away all but last / element
+poke_create_publish_file_step = steps.ShellCommand(
+ name="create publish file",
+ command=util.Interpolate('(echo -n gnupoke/; echo %(prop:branch)s | rev | cut -f1 -d/ | rev | sed -e s/master/trunk/) > /home/builder/shared/publish'))
+
# Upload automake test results, autoconf config.log, in both build and dist-build trees
elfutils_upload_to_bunsen = bunsen_logfile_upload_cpio_steps(
["*.trs", "*.log"])
@@ -4275,13 +4305,25 @@ systemtap_gentoo_sparc_builder = util.BuilderConfig(
c['builders'].append(systemtap_gentoo_sparc_builder)
# GNU poke builders and factories
-gnupoke_factory = util.BuildFactory()
-gnupoke_factory.addStep(steps.Git(
+
+# publish file based on branch name, cut away all but last / element
+poke_create_publish_file_step = steps.ShellCommand(
+ name="create publish file",
+ command=util.Interpolate('(echo -n gnupoke/; echo %(prop:branch)s | rev | cut -f1 -d/ | rev | sed -e s/master/trunk/) > /home/builder/shared/publish'))
+
+poke_create_output_step = steps.ShellCommand(
+ name="create output",
+ command="mkdir /home/builder/shared/output && mv ./poke-*.tar.gz /home/builder/shared/output/ && mv ./doc/poke.html /home/builder/shared/output/html && mv ./doc/poke.pdf /home/builder/shared/output/")
+
+poke_git_step = steps.Git(
repourl=gnupoke_repourl,
mode='full', method='fresh',
name="git checkout",
getDescription={'tags':True, 'always':True},
- haltOnFailure=True))
+ haltOnFailure=True)
+
+gnupoke_factory = util.BuildFactory()
+gnupoke_factory.addStep(poke_git_step)
gnupoke_factory.addStep(bootstrap_step)
gnupoke_factory.addStep(configure_step)
gnupoke_factory.addStep(make_step)
@@ -4291,6 +4333,19 @@ gnupoke_factory.addStep(steps.Test(command=["make", "syntax-check"],
haltOnFailure=False, flunkOnFailure=True))
gnupoke_factory.addSteps(bunsen_logfile_upload_cpio_steps(["*.sum", "*.log"]))
+# snapshots
+gnupoke_snapshots_factory = util.BuildFactory()
+gnupoke_snapshots_factory.addStep(poke_git_step)
+gnupoke_snapshots_factory.addStep(bootstrap_step)
+gnupoke_snapshots_factory.addStep(configure_step)
+gnupoke_snapshots_factory.addStep(make_step)
+gnupoke_snapshots_factory.addStep(make_html_step)
+gnupoke_snapshots_factory.addStep(make_pdf_step)
+gnupoke_snapshots_factory.addStep(make_dist_step)
+gnupoke_snapshots_factory.addStep(wait_snapshots_output_ready_step)
+gnupoke_snapshots_factory.addStep(poke_create_output_step)
+gnupoke_snapshots_factory.addStep(poke_create_publish_file_step)
+
gnupoke_debiantesting_x86_64_builder = util.BuilderConfig(
name="gnupoke-debian-testing-x86_64",
tags=["gnupoke", "debian-testing", "x86_64"],
@@ -4314,6 +4369,16 @@ gnupoke_gentoo_space_builder = util.BuilderConfig(
factory=gnupoke_factory)
c['builders'].append(gnupoke_gentoo_space_builder)
+poke_snapshots_builder = util.BuilderConfig(
+ name="poke-snapshots-html-pdf-dist",
+ collapseRequests=True,
+ properties={'container-file':
+ readContainerFile('fedora-latest')},
+ workernames="snapshots",
+ tags=["gnupoke-snapshots"],
+ factory=gnupoke_snapshots_factory)
+c['builders'].append(poke_snapshots_builder)
+
# annobin steps, builders and factories
# annobin checkouts have generated config files included, but might
--
2.31.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Add gnupoke snapshot scheduler, factory and builder
2023-04-16 19:05 [PATCH] Add gnupoke snapshot scheduler, factory and builder Mark Wielaard
@ 2023-04-16 19:53 ` Jose E. Marchesi
2023-04-16 20:50 ` Mark Wielaard
0 siblings, 1 reply; 5+ messages in thread
From: Jose E. Marchesi @ 2023-04-16 19:53 UTC (permalink / raw)
To: Mark Wielaard; +Cc: buildbot
Thanks Mark! :)
This and the Fedora ARM support means I should add new little graphics
to http://jemarch.net/poke?
> ---
> .../containers/Containerfile-fedora-latest | 1 +
> builder/master.cfg | 73 ++++++++++++++++++-
> 2 files changed, 70 insertions(+), 4 deletions(-)
>
> diff --git a/builder/containers/Containerfile-fedora-latest b/builder/containers/Containerfile-fedora-latest
> index e4016af..4fa217b 100644
> --- a/builder/containers/Containerfile-fedora-latest
> +++ b/builder/containers/Containerfile-fedora-latest
> @@ -13,6 +13,7 @@ RUN dnf upgrade -y && \
> jq json-c-devel socat \
> texinfo elfutils-debuginfod elfutils-debuginfod-client-devel \
> gcc-plugin-devel binutils-devel \
> + libnbd-devel gc-devel readline-devel texinfo-tex \
> xmlto xmlto-tex fop poppler-utils libxslt docbook-dtds docbook-style-xsl \
> libxml2-devel dos2unix dpkg koji python3-devel python3-koji python3-mock \
> python3-rpm python3-pyxdg mailcap openmpi-devel gdb help2man wget xxhash-devel \
> diff --git a/builder/master.cfg b/builder/master.cfg
> index 7564f58..fecdc76 100644
> --- a/builder/master.cfg
> +++ b/builder/master.cfg
> @@ -349,6 +349,15 @@ def master_and_users_try_branches(ref):
> return True
> return False
>
> +# For git maint branches (as gnu poke uses)
> +def master_and_maint_branches(ref):
> + if ref == "refs/heads/master":
> + return True
> + match = re.match ("refs/heads/maint/poke-[a-zA-Z0-9_-]+", ref)
> + if match:
> + return True
> + return False
> +
> sourceware_repourl='https://sourceware.org/git/builder.git'
> sourceware_gitpoller = changes.GitPoller(repourl=sourceware_repourl,
> branches=['main'],
> @@ -452,7 +461,7 @@ c['change_source'].append(glibc_gitpoller)
>
> gnupoke_repourl='https://git.savannah.gnu.org/git/poke.git'
> gnupoke_gitpoller = changes.GitPoller(repourl=gnupoke_repourl,
> - branches=['master'],
> + branches=master_and_maint_branches,
> pollInterval=3*60,
> pollRandomDelayMax=2*60,
> project='gnupoke')
> @@ -982,6 +991,16 @@ gnupoke_scheduler = schedulers.SingleBranchScheduler(
> "gnupoke-gentoo-sparc"])
> c['schedulers'].append(gnupoke_scheduler)
>
> +gnupoke_snapshot_scheduler = schedulers.Periodic(
> + name="gnupoke-snapshots",
> + change_filter=util.ChangeFilter(project="gnupoke",
> + branch_re=re.compile("master$|poke-[a-zA-Z0-9_-]+$")),
> + periodicBuildTimer=15*60, # 15 minutes in seconds
> + onlyIfChanged=True,
> + reason="gnupoke periodic project master/maint branch snapshot",
> + builderNames=["poke-snapshots-html-pdf-dist"])
> +c['schedulers'].append(gnupoke_snapshot_scheduler)
> +
> annobin_scheduler = schedulers.SingleBranchScheduler(
> name="annobin",
> change_filter=util.ChangeFilter(project="annobin",
> @@ -1082,6 +1101,12 @@ make_step = steps.Compile(
> util.Interpolate('-j%(prop:ncpus)s')]),
> name="make",
> haltOnFailure=True)
> +make_html_step = steps.Compile(
> + command=["make", "html"],
> + name="make html")
> +make_pdf_step = steps.Compile(
> + command=["make", "pdf"],
> + name="make pdf")
> make_check_step = steps.Test(
> command=addOutputSync.withArgs(["make", "check",
> util.Interpolate('-j%(prop:ncpus)s')]),
> @@ -1749,6 +1774,11 @@ elfutils_create_publish_file_step = steps.ShellCommand(
> name="create publish file",
> command="echo elfutils/coverage > /home/builder/shared/publish")
>
> +# publish file based on branch name, cut away all but last / element
> +poke_create_publish_file_step = steps.ShellCommand(
> + name="create publish file",
> + command=util.Interpolate('(echo -n gnupoke/; echo %(prop:branch)s | rev | cut -f1 -d/ | rev | sed -e s/master/trunk/) > /home/builder/shared/publish'))
> +
> # Upload automake test results, autoconf config.log, in both build and dist-build trees
> elfutils_upload_to_bunsen = bunsen_logfile_upload_cpio_steps(
> ["*.trs", "*.log"])
> @@ -4275,13 +4305,25 @@ systemtap_gentoo_sparc_builder = util.BuilderConfig(
> c['builders'].append(systemtap_gentoo_sparc_builder)
>
> # GNU poke builders and factories
> -gnupoke_factory = util.BuildFactory()
> -gnupoke_factory.addStep(steps.Git(
> +
> +# publish file based on branch name, cut away all but last / element
> +poke_create_publish_file_step = steps.ShellCommand(
> + name="create publish file",
> + command=util.Interpolate('(echo -n gnupoke/; echo %(prop:branch)s | rev | cut -f1 -d/ | rev | sed -e s/master/trunk/) > /home/builder/shared/publish'))
> +
> +poke_create_output_step = steps.ShellCommand(
> + name="create output",
> + command="mkdir /home/builder/shared/output && mv ./poke-*.tar.gz /home/builder/shared/output/ && mv ./doc/poke.html /home/builder/shared/output/html && mv ./doc/poke.pdf /home/builder/shared/output/")
> +
> +poke_git_step = steps.Git(
> repourl=gnupoke_repourl,
> mode='full', method='fresh',
> name="git checkout",
> getDescription={'tags':True, 'always':True},
> - haltOnFailure=True))
> + haltOnFailure=True)
> +
> +gnupoke_factory = util.BuildFactory()
> +gnupoke_factory.addStep(poke_git_step)
> gnupoke_factory.addStep(bootstrap_step)
> gnupoke_factory.addStep(configure_step)
> gnupoke_factory.addStep(make_step)
> @@ -4291,6 +4333,19 @@ gnupoke_factory.addStep(steps.Test(command=["make", "syntax-check"],
> haltOnFailure=False, flunkOnFailure=True))
> gnupoke_factory.addSteps(bunsen_logfile_upload_cpio_steps(["*.sum", "*.log"]))
>
> +# snapshots
> +gnupoke_snapshots_factory = util.BuildFactory()
> +gnupoke_snapshots_factory.addStep(poke_git_step)
> +gnupoke_snapshots_factory.addStep(bootstrap_step)
> +gnupoke_snapshots_factory.addStep(configure_step)
> +gnupoke_snapshots_factory.addStep(make_step)
> +gnupoke_snapshots_factory.addStep(make_html_step)
> +gnupoke_snapshots_factory.addStep(make_pdf_step)
> +gnupoke_snapshots_factory.addStep(make_dist_step)
> +gnupoke_snapshots_factory.addStep(wait_snapshots_output_ready_step)
> +gnupoke_snapshots_factory.addStep(poke_create_output_step)
> +gnupoke_snapshots_factory.addStep(poke_create_publish_file_step)
> +
> gnupoke_debiantesting_x86_64_builder = util.BuilderConfig(
> name="gnupoke-debian-testing-x86_64",
> tags=["gnupoke", "debian-testing", "x86_64"],
> @@ -4314,6 +4369,16 @@ gnupoke_gentoo_space_builder = util.BuilderConfig(
> factory=gnupoke_factory)
> c['builders'].append(gnupoke_gentoo_space_builder)
>
> +poke_snapshots_builder = util.BuilderConfig(
> + name="poke-snapshots-html-pdf-dist",
> + collapseRequests=True,
> + properties={'container-file':
> + readContainerFile('fedora-latest')},
> + workernames="snapshots",
> + tags=["gnupoke-snapshots"],
> + factory=gnupoke_snapshots_factory)
> +c['builders'].append(poke_snapshots_builder)
> +
> # annobin steps, builders and factories
>
> # annobin checkouts have generated config files included, but might
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Add gnupoke snapshot scheduler, factory and builder
2023-04-16 19:53 ` Jose E. Marchesi
@ 2023-04-16 20:50 ` Mark Wielaard
2023-04-17 9:29 ` Jose E. Marchesi
0 siblings, 1 reply; 5+ messages in thread
From: Mark Wielaard @ 2023-04-16 20:50 UTC (permalink / raw)
To: Jose E. Marchesi; +Cc: buildbot
Hi Jose,
On Sun, Apr 16, 2023 at 09:53:34PM +0200, Jose E. Marchesi wrote:
> This and the Fedora ARM support means I should add new little graphics
> to http://jemarch.net/poke?
For the fedora arm64 builder you could add:
<a href="https://builder.sourceware.org/buildbot/#builders/gnupoke-fedora-arm64"><img src="https://builder.sourceware.org/buildbot/badges/gnupoke-fedora-arm64.svg?left_text=%20%20fedora-arm64%20%20"></a>
For the snapshots we should see where they exactly land when there is
a real build. It was supposed to go into "trunk" or "poke-[123]" under
https://snapshots.sourceware.org/gnupoke/ but the first build didn't
have a branch associated with it, so it moved a level too high.
Please push a new commit to top of git or one of the maint/poke-*
branches, so we can check where they end up exactly.
Cheers,
Mark
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Add gnupoke snapshot scheduler, factory and builder
2023-04-16 20:50 ` Mark Wielaard
@ 2023-04-17 9:29 ` Jose E. Marchesi
2023-04-17 10:15 ` Mark Wielaard
0 siblings, 1 reply; 5+ messages in thread
From: Jose E. Marchesi @ 2023-04-17 9:29 UTC (permalink / raw)
To: Mark Wielaard; +Cc: buildbot, poke-devel
Hi Mark.
> On Sun, Apr 16, 2023 at 09:53:34PM +0200, Jose E. Marchesi wrote:
>> This and the Fedora ARM support means I should add new little graphics
>> to http://jemarch.net/poke?
>
> For the fedora arm64 builder you could add:
>
> <a
> href="https://builder.sourceware.org/buildbot/#builders/gnupoke-fedora-arm64"><img
> src="https://builder.sourceware.org/buildbot/badges/gnupoke-fedora-arm64.svg?left_text=%20%20fedora-arm64%20%20"></a>
I just added the above in https://jemarch.net/poke.
> For the snapshots we should see where they exactly land when there is
> a real build. It was supposed to go into "trunk" or "poke-[123]" under
> https://snapshots.sourceware.org/gnupoke/ but the first build didn't
> have a branch associated with it, so it moved a level too high.
>
> Please push a new commit to top of git or one of the maint/poke-*
> branches, so we can check where they end up exactly.
I just pushed a commit to `master'.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Add gnupoke snapshot scheduler, factory and builder
2023-04-17 9:29 ` Jose E. Marchesi
@ 2023-04-17 10:15 ` Mark Wielaard
0 siblings, 0 replies; 5+ messages in thread
From: Mark Wielaard @ 2023-04-17 10:15 UTC (permalink / raw)
To: Jose E. Marchesi; +Cc: buildbot, poke-devel
On Mon, 2023-04-17 at 11:29 +0200, Jose E. Marchesi wrote:
> Hi Mark.
>
> > On Sun, Apr 16, 2023 at 09:53:34PM +0200, Jose E. Marchesi wrote:
> > > This and the Fedora ARM support means I should add new little graphics
> > > to http://jemarch.net/poke?
> >
> > For the fedora arm64 builder you could add:
> >
> > <a
> > href="https://builder.sourceware.org/buildbot/#builders/gnupoke-fedora-arm64"><img
> > src="https://builder.sourceware.org/buildbot/badges/gnupoke-fedora-arm64.svg?left_text=%20%20fedora-arm64%20%20"></a>
>
> I just added the above in https://jemarch.net/poke.
And they are all green now \o/
> > For the snapshots we should see where they exactly land when there is
> > a real build. It was supposed to go into "trunk" or "poke-[123]" under
> > https://snapshots.sourceware.org/gnupoke/ but the first build didn't
> > have a branch associated with it, so it moved a level too high.
> >
> > Please push a new commit to top of git or one of the maint/poke-*
> > branches, so we can check where they end up exactly.
>
> I just pushed a commit to `master'.
Great, and that worked as expected, see
https://snapshots.sourceware.org/gnupoke/trunk/latest/
Cheers,
Mark
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-04-17 10:15 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-04-16 19:05 [PATCH] Add gnupoke snapshot scheduler, factory and builder Mark Wielaard
2023-04-16 19:53 ` Jose E. Marchesi
2023-04-16 20:50 ` Mark Wielaard
2023-04-17 9:29 ` Jose E. Marchesi
2023-04-17 10:15 ` Mark Wielaard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).