public inbox for buildbot@sourceware.org
 help / color / mirror / Atom feed
* glibc builder.sourceware.org integration
@ 2022-06-26 22:23 Mark Wielaard
  2022-07-01 12:00 ` Mark Wielaard
  0 siblings, 1 reply; 8+ messages in thread
From: Mark Wielaard @ 2022-06-26 22:23 UTC (permalink / raw)
  To: libc-alpha; +Cc: buildbot

Hi,

We have a shared buildbot for all Sourceware and GNU Toolchain
projects: https://builder.sourceware.org/
There are a couple of glibc builders now:
https://builder.sourceware.org/buildbot/#/builders?tags=glibc

The compile-only builders, glibc-debian-i386, glibc-debian-ppc64,
glibc-fedora-ppc64le, glibc-fedora-s390x, glibc-opensuseleap-x86_64
and glibc-opensusetw-x86_64 are all green. The builds are pretty quick
(using ccache) and take a couple of minutes at most. So they run on
each separate commit to the master branch.

It currently doesn't include arm64 and armhf builds, but those could
be added (the current boards are somewhat slow, but we will hopefully
get some faster hardware soon).

email notification of build failures hasn't been setup up yet. When we
set it up should it also notify the libc-alpha mailing list when a
build failure is detected? Only the first build failure will sent an
email till the build is restored.

There are currently two builders which do a make && make check.
glibc-fedora-x86_64 and glibc-fedrawhide-x86_64. The first is a
container build of fedora-latest (f36). The second is a fedora rawhide
VM.

The container builder has a couple of failures for which I just sent
patches. With those patches the build should turn green. At which
point we can also use it to detect test regressions.

Doing a full make && make check in the container takes ~15 to ~20
minutes and so isn't don't on every individual build, but batches up
changes till there have been none for 10 minutes. So it won't run
behind. It should detect a commit that caused a regression within 30
minutes. All test results (including all .out files) are uploaded into
the bunsendb for analysis:
https://builder.sourceware.org/testruns/?has_keyvalue_like_k=testrun.git_describe&has_keyvalue_like_v=%25glibc%25

Not all hardware is powerful enough to do a full make check on every
commit. But it would be good to make them do some real tests and not
just do a compile-only run. Is there a good, small, quick, sub-test
that could be added?

gdb and binutils are experimenting with git user try branches so they
can do pre-commit checking:
https://sourceware.org/pipermail/binutils/2022-June/121293.html
We can setup the same for glibc. Which we then can use for patchwork
integration.

Next week we should get more hardware and we can setup more full
builders for different configurations. Please contact
buildbot@sourceware.org with ideas and/or to help out with the setup
(which is in a public git repo, see the builder project homepage).

Cheers,

Mark


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: glibc builder.sourceware.org integration
  2022-06-26 22:23 glibc builder.sourceware.org integration Mark Wielaard
@ 2022-07-01 12:00 ` Mark Wielaard
  2022-07-01 22:24   ` Mark Wielaard
  0 siblings, 1 reply; 8+ messages in thread
From: Mark Wielaard @ 2022-07-01 12:00 UTC (permalink / raw)
  To: libc-alpha; +Cc: buildbot

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]

Hi all,

On Mon, Jun 27, 2022 at 12:23:55AM +0200, Mark Wielaard wrote:
> The compile-only builders, glibc-debian-i386, glibc-debian-ppc64,
> glibc-fedora-ppc64le, glibc-fedora-s390x, glibc-opensuseleap-x86_64
> and glibc-opensusetw-x86_64 are all green. The builds are pretty quick
> (using ccache) and take a couple of minutes at most. So they run on
> each separate commit to the master branch.

There is now also a glibc-rawhide-x86_64 builder.
https://builder.sourceware.org/buildbot/#/builders/glibc-rawhide-x86_64

> There are currently two builders which do a make && make check.
> glibc-fedora-x86_64 and glibc-fedrawhide-x86_64. The first is a
> container build of fedora-latest (f36). The second is a fedora rawhide
> VM.
> 
> The container builder has a couple of failures for which I just sent
> patches. With those patches the build should turn green. At which
> point we can also use it to detect test regressions.
> 
> Doing a full make && make check in the container takes ~15 to ~20
> minutes and so isn't don't on every individual build, but batches up
> changes till there have been none for 10 minutes. So it won't run
> behind. It should detect a commit that caused a regression within 30
> minutes. All test results (including all .out files) are uploaded into
> the bunsendb for analysis:
> https://builder.sourceware.org/testruns/?has_keyvalue_like_k=testrun.git_describe&has_keyvalue_like_v=%25glibc%25
> 
> Not all hardware is powerful enough to do a full make check on every
> commit. But it would be good to make them do some real tests and not
> just do a compile-only run. Is there a good, small, quick, sub-test
> that could be added?

For now I have added a make subdirs=elf -jncpus check step.  This
seems to have a couple of hundred tests which should be quick enough
to run on all builders. Test results are uploaded to the bunsendb for
comparison between distr/arches. Builders all seem to be green (except
for the full testrun builders of course), but the glibc-fedora-ppc64le builder has one FAIL:
FAIL: elf/tst-ldconfig-bad-aux-cache
https://builder.sourceware.org/buildbot/#/builders?tags=glibc
https://builder.sourceware.org/buildbot/#/builders/127/builds/35

Frank, I cannot find the corresponding bunsen test results for that
build under https://builder.sourceware.org/testruns/ did I make a
mistake in the bunsen upload step?

Cheers,

Mark

[-- Attachment #2: 0001-Add-make-subdirs-elf-jncpus-check-to-glibc_build_fac.patch --]
[-- Type: text/x-diff, Size: 2061 bytes --]

From 61c7d4849db2246f2792085e50e63e9ab3ea4c67 Mon Sep 17 00:00:00 2001
From: Mark Wielaard <mark@klomp.org>
Date: Fri, 1 Jul 2022 00:07:34 +0200
Subject: [PATCH] Add make subdirs=elf -jncpus check to glibc_build_factory

A full make check takes too long to run on all builder. But a make
subdirs=elf does run at least some tests in a couple of minutes. Also
upload results to bunsendb.

Factory is used on glibc-debian-i386, glibc-fedora-s390x,
glibc-debian-ppc64, glibc-fedora-ppc64le, glibc-opensusetw-x86_64,
glibc-opensuseleap-x86_64 and glibc-rawhide-x86_64.
---
 builder/master.cfg | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/builder/master.cfg b/builder/master.cfg
index 729ec37..e67df26 100644
--- a/builder/master.cfg
+++ b/builder/master.cfg
@@ -2224,6 +2224,15 @@ glibc_check_step = steps.Test(
         name='make check',
         haltOnFailure=False, flunkOnFailure=True)
 
+glibc_check_build_step = steps.Test(
+        workdir='glibc-build',
+        command=['make',
+                 'subdirs=elf',
+                 util.Interpolate('-j%(prop:ncpus)s'),
+                 'check'],
+        name='make check',
+        haltOnFailure=False, flunkOnFailure=True)
+
 glibc_bunsen_steps = bunsen_logfile_upload_cpio_steps(
         ["config.log", "*.out", "*.test-result", "*.sum"],
         workdir='../glibc-build')
@@ -2237,12 +2246,14 @@ glibc_factory.addStep(glibc_check_step)
 glibc_factory.addSteps(glibc_bunsen_steps)
 
 # A full check is too heavy-weight for some builders
-# Just do a build, no checks
+# Just do a build, only check one subdir (elf).
 glibc_build_factory = util.BuildFactory()
 glibc_build_factory.addStep(glibc_git_step)
 glibc_build_factory.addStep(glibc_rm_step)
 glibc_build_factory.addStep(glibc_configure_step)
 glibc_build_factory.addStep(glibc_make_build_step)
+glibc_build_factory.addStep(glibc_check_build_step)
+glibc_build_factory.addSteps(glibc_bunsen_steps)
 
 glibc_fedrawhide_x86_64_builder = util.BuilderConfig(
         name="glibc-fedrawhide-x86_64",
-- 
2.30.2


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: glibc builder.sourceware.org integration
  2022-07-01 12:00 ` Mark Wielaard
@ 2022-07-01 22:24   ` Mark Wielaard
  2022-07-02  1:09     ` Frank Ch. Eigler
  2022-07-04 15:17     ` Mark Wielaard
  0 siblings, 2 replies; 8+ messages in thread
From: Mark Wielaard @ 2022-07-01 22:24 UTC (permalink / raw)
  To: libc-alpha; +Cc: buildbot

Hi,

On Fri, Jul 01, 2022 at 02:00:42PM +0200, Mark Wielaard wrote:
> For now I have added a make subdirs=elf -jncpus check step.  This
> seems to have a couple of hundred tests which should be quick enough
> to run on all builders. Test results are uploaded to the bunsendb for
> comparison between distr/arches. Builders all seem to be green (except
> for the full testrun builders of course), but the glibc-fedora-ppc64le builder has one FAIL:
> FAIL: elf/tst-ldconfig-bad-aux-cache
> https://builder.sourceware.org/buildbot/#/builders?tags=glibc
> https://builder.sourceware.org/buildbot/#/builders/127/builds/35
> 
> Frank, I cannot find the corresponding bunsen test results for that
> build under https://builder.sourceware.org/testruns/ did I make a
> mistake in the bunsen upload step?

It is there now. I don't know why it wasn't there earlier.  Frank, is
there a delay between the bunsendb update and when the testruns show
up?

https://builder.sourceware.org/testrun/deda118447159c9affe00348cfb1ddf667563fe4

Which also contains the st-ldconfig-bad-aux-cache.out file

https://builder.sourceware.org/testrun/deda118447159c9affe00348cfb1ddf667563fe4?filename=elf%2Ftst-ldconfig-bad-aux-cache.out

Timed out: killed the child process

I couldn't immediately replicate that on a ppc64le setup.
Lets see if a next build also causes this test to time out.

Cheers,

Mark


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: glibc builder.sourceware.org integration
  2022-07-01 22:24   ` Mark Wielaard
@ 2022-07-02  1:09     ` Frank Ch. Eigler
  2022-07-02 19:45       ` Mark Wielaard
  2022-07-04 15:17     ` Mark Wielaard
  1 sibling, 1 reply; 8+ messages in thread
From: Frank Ch. Eigler @ 2022-07-02  1:09 UTC (permalink / raw)
  To: Mark Wielaard; +Cc: libc-alpha, buildbot

Hi -

> It is there now. I don't know why it wasn't there earlier.  Frank, is
> there a delay between the bunsendb update and when the testruns show
> up?

Yes.  Sourceware runs a 15-minute timer to ingest data from
bunsendb.git into the database.  (Yeah this could conceivably be done
by yet another buildbot job, but I'm happier with a distinctly
unprivileged process doing the work.)

- FChE

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: glibc builder.sourceware.org integration
  2022-07-02  1:09     ` Frank Ch. Eigler
@ 2022-07-02 19:45       ` Mark Wielaard
  2022-07-03 15:22         ` Dorian ROSSE
  0 siblings, 1 reply; 8+ messages in thread
From: Mark Wielaard @ 2022-07-02 19:45 UTC (permalink / raw)
  To: Frank Ch. Eigler; +Cc: buildbot, libc-alpha

Hi Frank,

On Fri, Jul 01, 2022 at 09:09:21PM -0400, Frank Ch. Eigler via Libc-alpha wrote:
> > It is there now. I don't know why it wasn't there earlier.  Frank, is
> > there a delay between the bunsendb update and when the testruns show
> > up?
> 
> Yes.  Sourceware runs a 15-minute timer to ingest data from
> bunsendb.git into the database.

Aha, ok, then it makes sense I didn't find it immediately.

BTW. For people who didn't see it yet, the builder frontpage
https://builder.sourceware.org/ contains little bunsen burner icons
next to the project or builder which are links to the recorded
testresults for that project or a particular builder so you can easily
dig into the results or compare results between bulds.

> (Yeah this could conceivably be done by yet another buildbot job,
> but I'm happier with a distinctly unprivileged process doing the
> work.)

Separation of processes/permissions is important. You can also do that
in the buildbot model, but it does take a bit more configuration than
just installing a cronjob. You could have a separate worker just for
the ingestion process. Then when the buildbot scheduler sees a change
in the bundendb.git it signals that worker to do a ingestion build
under its own process. Or you can use a Dependent or Triggerable
Scheduler to run a build against a different worker after a bunsen
upload step succeeds.

https://docs.buildbot.net/latest/manual/configuration/schedulers.html#dependent-scheduler
https://docs.buildbot.net/latest/manual/configuration/schedulers.html#sched-Triggerable

Cheers,

Mark


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: glibc builder.sourceware.org integration
  2022-07-02 19:45       ` Mark Wielaard
@ 2022-07-03 15:22         ` Dorian ROSSE
  2022-07-03 16:37           ` Mark Wielaard
  0 siblings, 1 reply; 8+ messages in thread
From: Dorian ROSSE @ 2022-07-03 15:22 UTC (permalink / raw)
  To: Mark Wielaard, Frank Ch. Eigler; +Cc: buildbot, libc-alpha

[-- Attachment #1: Type: text/plain, Size: 2467 bytes --]

Hello,


How to build on Ubuntu?

I have ever tried without success the documentation doesn't bring the good help,

This is for build systemd version 251,

Each help will be good,

Now the server doesn't power up fully because it has error " userspace out of memory killer" thus I will bought ram,

I have this error because I have uninstall xcb, Wayland and libbluetooth3,

Thanks you in advance for your help,

Regards.


Dorian Rosse.
________________________________
From: Libc-alpha <libc-alpha-bounces+dorianbrice=hotmail.fr@sourceware.org> on behalf of Mark Wielaard <mark@klomp.org>
Sent: Saturday, July 2, 2022 9:45:49 PM
To: Frank Ch. Eigler <fche@elastic.org>
Cc: buildbot@sourceware.org <buildbot@sourceware.org>; libc-alpha@sourceware.org <libc-alpha@sourceware.org>
Subject: Re: glibc builder.sourceware.org integration

Hi Frank,

On Fri, Jul 01, 2022 at 09:09:21PM -0400, Frank Ch. Eigler via Libc-alpha wrote:
> > It is there now. I don't know why it wasn't there earlier.  Frank, is
> > there a delay between the bunsendb update and when the testruns show
> > up?
>
> Yes.  Sourceware runs a 15-minute timer to ingest data from
> bunsendb.git into the database.

Aha, ok, then it makes sense I didn't find it immediately.

BTW. For people who didn't see it yet, the builder frontpage
https://builder.sourceware.org/ contains little bunsen burner icons
next to the project or builder which are links to the recorded
testresults for that project or a particular builder so you can easily
dig into the results or compare results between bulds.

> (Yeah this could conceivably be done by yet another buildbot job,
> but I'm happier with a distinctly unprivileged process doing the
> work.)

Separation of processes/permissions is important. You can also do that
in the buildbot model, but it does take a bit more configuration than
just installing a cronjob. You could have a separate worker just for
the ingestion process. Then when the buildbot scheduler sees a change
in the bundendb.git it signals that worker to do a ingestion build
under its own process. Or you can use a Dependent or Triggerable
Scheduler to run a build against a different worker after a bunsen
upload step succeeds.

https://docs.buildbot.net/latest/manual/configuration/schedulers.html#dependent-scheduler
https://docs.buildbot.net/latest/manual/configuration/schedulers.html#sched-Triggerable

Cheers,

Mark


[-- Attachment #2: Type: text/html, Size: 5354 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: glibc builder.sourceware.org integration
  2022-07-03 15:22         ` Dorian ROSSE
@ 2022-07-03 16:37           ` Mark Wielaard
  0 siblings, 0 replies; 8+ messages in thread
From: Mark Wielaard @ 2022-07-03 16:37 UTC (permalink / raw)
  To: Dorian ROSSE; +Cc: Frank Ch. Eigler, buildbot, libc-alpha

Hi Dorian,

On Sun, Jul 03, 2022 at 03:22:54PM +0000, Dorian ROSSE wrote:
> How to build on Ubuntu?

The buildbot currently doesn't have a Ubuntu builder, but it does do a
Debian build of glibc on various architectures. If you have the
packages installed that the buildbot Debian container has, then it
should work the same as the Debian glibc builder:
https://sourceware.org/git/?p=builder.git;a=blob;f=builder/containers/Containerfile-debian-stable;hb=HEAD
https://sourceware.org/git/?p=builder.git;a=blob;f=builder/master.cfg;hb=HEAD#l2277

If you are interested in rebuilding the Debian (or Ubuntu) glibc
package then maybe this tutorial will be helpful:
https://wiki.debian.org/BuildingTutorial

Cheers,

Mark


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: glibc builder.sourceware.org integration
  2022-07-01 22:24   ` Mark Wielaard
  2022-07-02  1:09     ` Frank Ch. Eigler
@ 2022-07-04 15:17     ` Mark Wielaard
  1 sibling, 0 replies; 8+ messages in thread
From: Mark Wielaard @ 2022-07-04 15:17 UTC (permalink / raw)
  To: libc-alpha; +Cc: buildbot

Hi,

On Sat, 2022-07-02 at 00:24 +0200, Mark Wielaard wrote:
> On Fri, Jul 01, 2022 at 02:00:42PM +0200, Mark Wielaard wrote:
> > For now I have added a make subdirs=elf -jncpus check step.  This
> > seems to have a couple of hundred tests which should be quick
> > enough
> > to run on all builders. Test results are uploaded to the bunsendb
> > for
> > comparison between distr/arches. Builders all seem to be green
> > (except
> > for the full testrun builders of course), but the glibc-fedora-
> > ppc64le builder has one FAIL:
> > FAIL: elf/tst-ldconfig-bad-aux-cache
> > https://builder.sourceware.org/buildbot/#/builders?tags=glibc
> > https://builder.sourceware.org/buildbot/#/builders/127/builds/35
> > 
> > Frank, I cannot find the corresponding bunsen test results for that
> > build under https://builder.sourceware.org/testruns/ did I make a
> > mistake in the bunsen upload step?
> 
> It is there now. I don't know why it wasn't there earlier.  Frank, is
> there a delay between the bunsendb update and when the testruns show
> up?
> https://builder.sourceware.org/testrun/deda118447159c9affe00348cfb1ddf667563fe4
> 
> Which also contains the tst-ldconfig-bad-aux-cache.out file
> https://builder.sourceware.org/testrun/deda118447159c9affe00348cfb1ddf667563fe4?filename=elf%2Ftst-ldconfig-bad-aux-cache.out
> 
> Timed out: killed the child process
> 
> I couldn't immediately replicate that on a ppc64le setup.
> Lets see if a next build also causes this test to time out.

It was only that one build. All others are green:
https://builder.sourceware.org/buildbot/#/builders/glibc-fedora-ppc64le

I haven't figured out or have been able to replicate this issue. The
test runs ldconfig with a corrupt aux-cache and checks it doesn't
crash. Maybe instead of crashing sometimes ldconfig goes into a loop?

Cheers,

Mark

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2022-07-04 15:17 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-26 22:23 glibc builder.sourceware.org integration Mark Wielaard
2022-07-01 12:00 ` Mark Wielaard
2022-07-01 22:24   ` Mark Wielaard
2022-07-02  1:09     ` Frank Ch. Eigler
2022-07-02 19:45       ` Mark Wielaard
2022-07-03 15:22         ` Dorian ROSSE
2022-07-03 16:37           ` Mark Wielaard
2022-07-04 15:17     ` Mark Wielaard

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).