Re: [PATCH builder.git] master.cfg: add GNU poke builder

["Jose E. Marchesi" <jemarch@gnu.org>]

Hi Arsen.
Thank you so much for working on this.

> ---
> Evening,
>
> This patch (when finished ;) will add a GNU poke builder to the Sourceware
> buildbot.
>
> There's a few things to consider before merging this (CC'd poke-devel@ for the
> discussion):
> - What distros and CPUs do we want to test on?
>
>   I think at least s390x in conjunction with x86_64 would be useful, to provide
>   a reference for big endian machines, to detect if we're accidentally relying
>   on endianness.
>
>   It would probably be useful to test on Debian stable+testing and Fedora
>   latest+rawhide, just to make sure we work in both in the past and in the
>   future; though, it's probably less significant than CPU choices.
>
>   I only left debian-testing in in the current revision of the patch, which has
>   UNSUPPORTED tests due to libtextstyle.

Regarding CPUs, Jitter is quite sensible to them when run in
non-threaded mode, so it is a good thing to test on them in order to
detect issues before releases.

For example, we had several problems with (if I remember properly) arm
32-bit and some other arch with poke 2.0.

Mohammad and Luca may have particular suggestions..

> - Which check do we run?
>
>   I initially tried running distcheck, but automake would delete the resulting
>   test run AFAICT.  I'm not entirely sure how to work around that yet (and the
>   night is catching up with me at this point).  As a result, I switched to just
>   check.  Would distcheck even benefit us on CI?  My current thinking is "maybe
>   not", and it does result in compiling poke twice, FWIW.

As discussed in IRC, this is the script that Bruno uses in the gitlab
CI.  I think replicating it may be good:

https://gitlab.com/gnu-poke/ci-distcheck

>
> ... and, of course, the rest of the patch is up for discussion ;)

FWIW it looks good to me.  We are gonna need some documenation for this
in etc/hacking.org regarding this though :)

Thanks again!

>
> Thank you in advance, and have a wonderful night!
>
>  README                                        |  2 +-
>  .../containers/Containerfile-debian-testing   |  1 +
>  builder/master.cfg                            | 65 ++++++++++++++++++-
>  3 files changed, 66 insertions(+), 2 deletions(-)
>
> diff --git a/README b/README
> index d2b04cb..2f42d2d 100644
> --- a/README
> +++ b/README
> @@ -12,7 +12,7 @@ htdocs: document root of https://builder.sourceware.org/
>    - Define a repourl to be used in the GitPoller and any Build Steps
>      project_repourl = 'git://sourceware.org/git/project.git'
>  
> -  - If the repourl is "new" (not sourceware, gcc or wildebeest)
> +  - If the repourl is "new" (not sourceware, gcc, savannah, or wildebeest)
>      then add a revlink matches (at the end of master.cfg) so
>      changesets get an URL to the actual commit.
>  
> diff --git a/builder/containers/Containerfile-debian-testing b/builder/containers/Containerfile-debian-testing
> index 31a62cb..793cbe8 100644
> --- a/builder/containers/Containerfile-debian-testing
> +++ b/builder/containers/Containerfile-debian-testing
> @@ -18,6 +18,7 @@ RUN apt-get update && \
>        libxml2-dev dos2unix python3-dev python3-mock \
>        python3-rpm python3-xdg mime-support libopenmpi-dev gdb help2man wget libxxhash-dev \
>        libjson-c-dev jq socat \
> +      libgc-dev libreadline-dev libnbd-dev \
>        buildbot-worker && \
>      apt-get clean
>  
> diff --git a/builder/master.cfg b/builder/master.cfg
> index 2bc46ef..48ffbf0 100644
> --- a/builder/master.cfg
> +++ b/builder/master.cfg
> @@ -426,6 +426,13 @@ glibc_gitpoller = changes.GitPoller(repourl=glibc_repourl,
>                                      project='glibc')
>  c['change_source'].append(glibc_gitpoller)
>  
> +gnupoke_repourl='https://git.savannah.gnu.org/git/poke.git'
> +gnupoke_gitpoller = changes.GitPoller(repourl=gnupoke_repourl,
> +                                      branches=['master'],
> +                                      pollInterval=3*60,
> +                                      pollRandomDelayMax=2*60,
> +                                      project='gnupoke')
> +c['change_source'].append(gnupoke_gitpoller)
>  
>  
>  
> @@ -868,6 +875,12 @@ glibc_build_scheduler = schedulers.SingleBranchScheduler(
>                        "glibc-opensuseleap-x86_64"])
>  c['schedulers'].append(glibc_build_scheduler)
>  
> +gnupoke_scheduler = schedulers.SingleBranchScheduler(
> +        name="gnupoke",
> +        change_filter=util.ChangeFilter(project="gnupoke",
> +                                        branch="master"),
> +        builderNames=["gnupoke-debian-testing-x86_64"])
> +c['schedulers'].append(gnupoke_scheduler)
>  
>  ####### BUILDERS
>  
> @@ -911,6 +924,9 @@ sourceware_reconfig_builder = util.BuilderConfig(
>  c['builders'].append(sourceware_reconfig_builder)
>  
>  # Some common build steps
> +bootstrap_step = steps.ShellCommand(
> +        command=["./bootstrap"],
> +        name="bootstrap", haltOnFailure=True)
>  autoreconf_step = steps.ShellCommand(
>          command=["autoreconf", "-f", "-i"],
>          name="autoreconf", haltOnFailure=True)
> @@ -3739,6 +3755,31 @@ systemtap_fedrawhide_x86_64_builder = util.BuilderConfig(
>          factory=systemtap_factory)
>  c['builders'].append(systemtap_fedrawhide_x86_64_builder)
>  
> +# GNU poke builders and factories
> +gnupoke_factory = util.BuildFactory()
> +gnupoke_factory.addStep(steps.Git(
> +        repourl=gnupoke_repourl,
> +        mode='full', method='fresh',
> +        name="git checkout",
> +        haltOnFailure=True))
> +gnupoke_factory.addStep(bootstrap_step)
> +gnupoke_factory.addStep(configure_step)
> +gnupoke_factory.addStep(make_step)
> +gnupoke_factory.addStep(make_distcheck_step)
> +gnupoke_factory.addSteps(bunsen_logfile_upload_steps([
> +        (["./testsuite/poke.log"], "testsuite"),
> +        (["config.log"],           "."),
> +        (["./jitter/config.log"],  "jitter")
> +]))
> +
> +gnupoke_debiantesting_x86_64_builder = util.BuilderConfig(
> +        name="gnupoke-debian-testing-x86_64",
> +        tags=["gnupoke", "debian-testing", "x86_64"],
> +        properties={'container-file':
> +                    readContainerFile('debian-testing')},
> +        workernames=vm_workers,
> +        factory=gnupoke_factory)
> +c['builders'].append(gnupoke_debiantesting_x86_64_builder)
>  
>  ####### BUILDBOT SERVICES
>  
> @@ -4000,6 +4041,25 @@ mn_valgrind_change = reporters.MailNotifier(
>          generators=[generator_valgrind_change])
>  c['services'].append(mn_valgrind_change)
>  
> +# Change reporter for GNU poke.
> +generator_gnupoke = reporters.BuildSetStatusGenerator(
> +        mode=('change',), tags=['gnupoke'])
> +mn_gnupoke = reporters.MailNotifier(
> +        fromaddr="builder@sourceware.org",
> +        sendToInterestedUsers=False,
> +        extraRecipients=['poke-devel@gnu.org'],
> +        generators=[generator_gnupoke])
> +c['services'].append(mn_gnupoke)
> +
> +# Problem reporter for GNU poke.
> +generator_gnupoke_problem = reporters.BuildSetStatusGenerator(
> +        mode=('problem',), tags=['gnupoke'])
> +mn_gnupoke_problem = reporters.MailNotifier(
> +        fromaddr="builder@sourceware.org",
> +        sendToInterestedUsers=True,
> +        generators=[generator_gnupoke_problem])
> +c['services'].append(mn_gnupoke_problem)
> +
>  
>  ####### PROJECT IDENTITY
>  
> @@ -4071,8 +4131,11 @@ wildebeestRevLink = util.RevlinkMatch(
>          repo_urls=[r'git://code.wildebeest.org/(.*).git',
>                     r'https://code.wildebeest.org/git/(.*)'],
>          revlink=r'https://code.wildebeest.org/git/\1/commit/?id=%s')
> +savannahRevLink = util.RevlinkMatch(
> +        repo_urls=[r'https://git.savannah.gnu.org/git/(.*).git'],
> +        revlink=r'https://git.savannah.gnu.org/cgit/\1.git/commit/?id=%s')
>  
> -reflinks = [sourcewareRevLink, gccRevLink, wildebeestRevLink]
> +reflinks = [sourcewareRevLink, gccRevLink, wildebeestRevLink, savannahRevLink]
>  
>  def revlinkmultiplexer(rev,repo):
>    for revlink in reflinks: