public inbox for gdb@sourceware.org
 help / color / mirror / Atom feed
* buildbot users try branch builders
@ 2022-06-20 10:48 Mark Wielaard
  2022-06-21 10:30 ` Szabolcs Nagy
  2022-06-22  6:32 ` Jan Beulich
  0 siblings, 2 replies; 9+ messages in thread
From: Mark Wielaard @ 2022-06-20 10:48 UTC (permalink / raw)
  To: binutils, gdb; +Cc: buildbot, Thomas Fitzsimmons, fche

Hi,

As an experiment we now have try builders for binutils and gdb. If you
have commit access to binutils-gdb.git you can now push to your
users/<name>/try-xxx branch and the buildbot will do builds and sent
you (the commit author) email about the results. The builds are also
visible at:

https://builder.sourceware.org/buildbot/#/builders?tags=binutils-try
https://builder.sourceware.org/buildbot/#/builders?tags=gdb-try

My workflow to use this looks like:
- git checkout -b frob
- hack, hack, hack... OK, looks good to submit
- git commit -a -m "Awesome hack"
- git push origin frob:users/mark/try-frob
- ... wait for the emails to come in or watch buildbot logs ...
- Send in patches and mention what the try bot reported

After your patches have been accepted you can delete the branch again:
- git push origin :users/mark/try-frob

Please use this only for patches you intend to submit and think are
ready but want to double check. Use the GCC Compile Farm machines if
you need to hack edit/compile/debug on a specific architecture. But if
you really need access to one of the buildbot machines for
investigating an issue, please contact buildbot@sourceware.org.

The armhf and arm64 builders have left out for now since they are a
bit too slow. We'll hopefully get a 8 core Ampare machine from the
Oregon State University this week so we can also enable those arm
builders.

Tom, I haven't hooked up the debian-ppc64 builder yet. It would take
another ~3GB disk space for the new try builders.

Frank, the try builders also upload logs to the bunsendb. And so are
visible here https://builder.sourceware.org/testruns/
I think this is useful, but maybe these try logs are just noise?

Please sent feedback (bad or good) to buildbot@sourceware.org

Currently this is only for the binutils and gdb projects, but if this
looks useful and when we get some new hardware we can roll it out to
other sourceware projects.

Cheers,

Mark


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: buildbot users try branch builders
  2022-06-20 10:48 buildbot users try branch builders Mark Wielaard
@ 2022-06-21 10:30 ` Szabolcs Nagy
  2022-06-21 17:30   ` Mark Wielaard
  2022-06-22  6:32 ` Jan Beulich
  1 sibling, 1 reply; 9+ messages in thread
From: Szabolcs Nagy @ 2022-06-21 10:30 UTC (permalink / raw)
  To: Mark Wielaard; +Cc: binutils, gdb, buildbot, Thomas Fitzsimmons, fche

The 06/20/2022 12:48, Mark Wielaard wrote:
> The armhf and arm64 builders have left out for now since they are a
> bit too slow. We'll hopefully get a 8 core Ampare machine from the
> Oregon State University this week so we can also enable those arm
> builders.

what is the requirement for the machine? i have an aarch64 machine
for public gnu tools ci, and i can set up build slaves there if
there are instructions how to do it.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: buildbot users try branch builders
  2022-06-21 10:30 ` Szabolcs Nagy
@ 2022-06-21 17:30   ` Mark Wielaard
  0 siblings, 0 replies; 9+ messages in thread
From: Mark Wielaard @ 2022-06-21 17:30 UTC (permalink / raw)
  To: Szabolcs Nagy; +Cc: binutils, gdb, buildbot, Thomas Fitzsimmons, fche

Hi Szabolcs,

On Tue, 2022-06-21 at 11:30 +0100, Szabolcs Nagy wrote:
> The 06/20/2022 12:48, Mark Wielaard wrote:
> > The armhf and arm64 builders have left out for now since they are a
> > bit too slow. We'll hopefully get a 8 core Ampare machine from the
> > Oregon State University this week so we can also enable those arm
> > builders.
> 
> what is the requirement for the machine? i have an aarch64 machine
> for public gnu tools ci, and i can set up build slaves there if
> there are instructions how to do it.

The requirements are not that big, in theory that tinker-board and
odroid-board are fine as buildbot-workers. But only for smaller builds.
To actually run some of the larger builds it should have at least 8GB
and 4cpus. More is better of course. Especially for the larger projects
if you want to do full test runs and not just "quick" CI builds.

Disk again depends on how many builds/projects you want to support. It
is really beneficial to have a bigger ~5GB ccache for repeat builds.
binutils takes ~2G (plus another ~2G for the try builder), similar for
gdb and libabigail, gccrust is ~4.5G, valgrind ~600MB, elfutils ~300MB,
bzip2, dwz are just a couple of MB. If we want to add glibc and enable
try builders for more projects I estimate 30GB to 50GB. The gccrust
builder does just a --disable-bootstrap build, so a full gcc builder
would add another 10G at least (maybe 2 or 3 times if we want to add
try builders). If you like to do container builds then those are almost
full distros, so add at least another 4GB per container/distro type (we
currently support 4). So you can get away with ~15GB, but might need up
to ~300GB if you want to do more and larger builds across multiple
distros.

There is a little documentation on adding workers (they used to be
called slaves, but buildbot renamed them some time ago) in the README:
https://sourceware.org/git/?p=builder.git;a=blob;f=README;hb=HEAD#l111
But that is mostly from the side of the buildbot master.

The easiest setup is one with a dedicated distro install with a
buildbot-worker: 
https://docs.buildbot.net/current/manual/installation/worker.html

All you really need to do is make sure a full distro is installed with
all development packages. You can take a peek at one of the container
files to see which: 
https://sourceware.org/git/?p=builder.git;a=tree;f=builder/containers;hb=HEAD
Then create a user which can run the buildbot-worker.
Please contact the buildbot@sourceware.org mailinglist for connection
info.

Another way is making sure you have docker installed (podman sadly
seems incompatible at the moment) and a builder user that is in the
docker group which can be accessed through ssh to create full container
images. But that is a bit more work (and currently depends on some
specific user uid to make in/out container sharing easier). Please
again contact the buildbot@sourceware.org mailinglist for more info.

There are several people on the list who have setup buildbot-workers
who can help. And then we can coordinate which projects/builders you
want to support. https://sourceware.org/mailman/listinfo/buildbot

Cheers,

Mark

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: buildbot users try branch builders
  2022-06-20 10:48 buildbot users try branch builders Mark Wielaard
  2022-06-21 10:30 ` Szabolcs Nagy
@ 2022-06-22  6:32 ` Jan Beulich
  2022-06-22  7:30   ` Andreas Schwab
  2022-06-22 10:51   ` Mark Wielaard
  1 sibling, 2 replies; 9+ messages in thread
From: Jan Beulich @ 2022-06-22  6:32 UTC (permalink / raw)
  To: Mark Wielaard; +Cc: buildbot, Thomas Fitzsimmons, fche, binutils, gdb

On 20.06.2022 12:48, Mark Wielaard wrote:
> As an experiment we now have try builders for binutils and gdb. If you
> have commit access to binutils-gdb.git you can now push to your
> users/<name>/try-xxx branch and the buildbot will do builds and sent
> you (the commit author) email about the results. The builds are also
> visible at:

So I have a (perhaps seemingly unrelated) question here: If this is put
into general use, it'll encourage more people to push to such branches.
I'm not very fluent in git, yet before that might happen I'd like to
figure a way to keep my clone clean of the objects related to such
branches, but without also omitting the objects for, in particular,
release branches. Hence in .git/config

	fetch = +refs/heads/*:refs/remotes/origin/*

is too wide while limiting to just master is too narrow. From earlier
experiments I seem to recall that the *s here can't be simply replaced
by more restricting wildcard "expressions" ...

Jan

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: buildbot users try branch builders
  2022-06-22  6:32 ` Jan Beulich
@ 2022-06-22  7:30   ` Andreas Schwab
  2022-06-22 10:51   ` Mark Wielaard
  1 sibling, 0 replies; 9+ messages in thread
From: Andreas Schwab @ 2022-06-22  7:30 UTC (permalink / raw)
  To: Jan Beulich via Binutils
  Cc: Mark Wielaard, Jan Beulich, gdb, buildbot, Thomas Fitzsimmons, fche

On Jun 22 2022, Jan Beulich via Binutils wrote:

> release branches. Hence in .git/config
>
> 	fetch = +refs/heads/*:refs/remotes/origin/*
>
> is too wide while limiting to just master is too narrow.

With the latest git version, you can have negative refspecs.

           If a refspec is prefixed by ^, it will be interpreted as a negative
           refspec. Rather than specifying which refs to fetch or which local
           refs to update, such a refspec will instead specify refs to
           exclude. A ref will be considered to match if it matches at least
           one positive refspec, and does not match any negative refspec.
           Negative refspecs can be useful to restrict the scope of a pattern
           refspec so that it will not include specific refs. Negative
           refspecs can themselves be pattern refspecs. However, they may only
           contain a <src> and do not specify a <dst>. Fully spelled out hex
           object names are also not supported.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: buildbot users try branch builders
  2022-06-22  6:32 ` Jan Beulich
  2022-06-22  7:30   ` Andreas Schwab
@ 2022-06-22 10:51   ` Mark Wielaard
  2022-06-22 11:04     ` Andreas Schwab
  1 sibling, 1 reply; 9+ messages in thread
From: Mark Wielaard @ 2022-06-22 10:51 UTC (permalink / raw)
  To: Jan Beulich; +Cc: buildbot, Thomas Fitzsimmons, fche, binutils, gdb

Hi Jan,

On Wed, 2022-06-22 at 08:32 +0200, Jan Beulich wrote:
> I'm not very fluent in git, yet before that might happen I'd like to
> figure a way to keep my clone clean of the objects related to such
> branches, but without also omitting the objects for, in particular,
> release branches. Hence in .git/config
> 
> 	fetch = +refs/heads/*:refs/remotes/origin/*
> 
> is too wide while limiting to just master is too narrow. From earlier
> experiments I seem to recall that the *s here can't be simply
> replaced by more restricting wildcard "expressions" ...

Note that you can have multiple fetch lines, so you can just add a list
of explicit refs you want to fetch.

But I do see your problem, unlike in gcc where the releases are all
under refs/heads/releases/ the binutils and gdb ones are directly under
/heads/.

Maybe the negative refspecs that Andreas mentioned would help here
because all user branches are under /ref/heads/users/ So maybe adding
fetch = ^refs/heads/users/* would do the trick? (untested)

Cheers,

Mark

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: buildbot users try branch builders
  2022-06-22 10:51   ` Mark Wielaard
@ 2022-06-22 11:04     ` Andreas Schwab
  2022-07-04  6:39       ` Jan Beulich
  0 siblings, 1 reply; 9+ messages in thread
From: Andreas Schwab @ 2022-06-22 11:04 UTC (permalink / raw)
  To: Mark Wielaard
  Cc: Jan Beulich, gdb, buildbot, Thomas Fitzsimmons, fche, binutils

On Jun 22 2022, Mark Wielaard wrote:

> Maybe the negative refspecs that Andreas mentioned would help here
> because all user branches are under /ref/heads/users/ So maybe adding
> fetch = ^refs/heads/users/* would do the trick? (untested)

The user branches are under a different name space (refs/users).  Only
refs/heads/devel is matched in addition to the release and master
branches.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: buildbot users try branch builders
  2022-06-22 11:04     ` Andreas Schwab
@ 2022-07-04  6:39       ` Jan Beulich
  2022-07-04 14:10         ` Andreas Schwab
  0 siblings, 1 reply; 9+ messages in thread
From: Jan Beulich @ 2022-07-04  6:39 UTC (permalink / raw)
  To: Andreas Schwab
  Cc: gdb, buildbot, Thomas Fitzsimmons, fche, binutils, Mark Wielaard

On 22.06.2022 13:04, Andreas Schwab wrote:
> On Jun 22 2022, Mark Wielaard wrote:
> 
>> Maybe the negative refspecs that Andreas mentioned would help here
>> because all user branches are under /ref/heads/users/ So maybe adding
>> fetch = ^refs/heads/users/* would do the trick? (untested)
> 
> The user branches are under a different name space (refs/users).  Only
> refs/heads/devel is matched in addition to the release and master
> branches.

I don't speak enough git to be able to tell in how far it is related,
but looking at just the output of a pull done a minute ago I see that
it also fetched

   bf5ddcecc07c..f0a8e7c6fed0  users/ARM/morello-binutils-gdb-master -> origin/users/ARM/morello-binutils-gdb-master

Perhaps I misread your reply by interpreting it to mean Mark's
suggestion won't make a difference, and hence a negative fetch line
like suggested by is pointless to add (for being default behavior
anyway)? (I didn't add or alter any of the fetch lines yet, I'm
merely still in the process of figuring what exactly I need to
change such that it would have the intended effect of not fetching
anything that's under users/.)

Jan

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: buildbot users try branch builders
  2022-07-04  6:39       ` Jan Beulich
@ 2022-07-04 14:10         ` Andreas Schwab
  0 siblings, 0 replies; 9+ messages in thread
From: Andreas Schwab @ 2022-07-04 14:10 UTC (permalink / raw)
  To: Jan Beulich via Gdb
  Cc: Jan Beulich, buildbot, Thomas Fitzsimmons, fche, binutils, Mark Wielaard

On Jul 04 2022, Jan Beulich via Gdb wrote:

> it also fetched
>
>    bf5ddcecc07c..f0a8e7c6fed0  users/ARM/morello-binutils-gdb-master -> origin/users/ARM/morello-binutils-gdb-master

Sorry, I had mixed it up with the gcc repository, where the user
branches are located on the non-default refs/users namespace.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2022-07-04 14:10 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-20 10:48 buildbot users try branch builders Mark Wielaard
2022-06-21 10:30 ` Szabolcs Nagy
2022-06-21 17:30   ` Mark Wielaard
2022-06-22  6:32 ` Jan Beulich
2022-06-22  7:30   ` Andreas Schwab
2022-06-22 10:51   ` Mark Wielaard
2022-06-22 11:04     ` Andreas Schwab
2022-07-04  6:39       ` Jan Beulich
2022-07-04 14:10         ` Andreas Schwab

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).