[GDB BuildBot] New "Try Server"

[GDB BuildBot] New "Try Server"

[Sergio Durigan Junior]

[-- Attachment #1: Type: text/plain, Size: 9967 bytes --]

Hi there,

0. Introduction
===============

I am happy to announce that I have finally finished configuring our
BuildBot instance to accept "try jobs".  What this means is that a
developer will be able to upload a patch to our BuildBot, and this patch
will then be tested against a set of builders we have there.  In the
paragraphs below I will explain how this feature works, who can use it
(and how), and try to answer a few questions that I think you will have.

1. Concepts
===========

As you know, our BuildBot master runs on my personal server, and we have
several "slaves" running on different locations.  The master doesn't do
any builds itself, but rather controls all the details about the builds
that need to be performed.  The slaves do not take any decision by
themselves; they just do what the master tells them to do.

On top of the slaves, there is an abstraction called "builder".  The
builder is basically a recipe of how to do a build.  This includes the
compilation flags used, environment variables, and testsuite options.  A
slave can have multiple builders; a builder can also be assigned to
multiple slaves (for example, when we have more than one slave with the
same hardware configuration, as is the case of the Fedora x86_64
slaves).

Then, we have the schedulers.  They are responsible for monitoring
events and triggering builds.  Our main scheduler monitors our
repository, and triggers a full build on all builders whenever a new
commit has been pushed.  We also have "cron job" schedulers, used to
trigger racy testcase detection builds every week at a certain time.
And now, we have the "try job" scheduler, responsible for triggering a
build on a set of builders when a developer has submitted a patch for
testing.

2. Build priority
=================

Before the try scheduler, builds were basically treated in a "first
come, first served" basis.  No special priority was assigned to a build.

Now, I decided to try a new algorithm.  Considering that a developer
usually want to test patches when they are almost ready for submission,
it would be really annoying if she had to wait in line until her build
is processed; it is not rare to see some builders having 10 or more
builds in the queue.  So, my decision was to "move" the try builds to
the top of the queue.  This will obviously delay the testing of the
official repository, but I am hoping that the impact will not be very
big.  If it is, then this decision will have to be revisited.

3. Security
===========

I am currently talking to the buildslave owners off-list in order to
make sure that we are on the same page here, but there are a few
considerations to be made about the security of a try job.  The obvious
one is that a developer may upload a malicious patch to be "tested", and
there is no easy way for BuildBot to detect this.  Since the BuildBot
master does not build anything, it would be the slaves who would suffer
From this kind of attack.

But there is another thing to consider.  In order to be able to upload a
patch to be tested, the developer must have some kind of access to the
BuildBot master.  Yes, my personal server.  No, I am not happy about
this.  I will write more on the authentication mechanism below.

So, after giving a long thought about this problem, I decided that I
will only provide accounts to:

  - Global maintainers.

  - Someone I know personally, or that I feel confident enough that is a
    trustworthy person.  (yeah, this is very subjective, but it is my
    machine :-/)

If you do not fit into one of these categories, you can ask someone from
one of the above categories to vouch for you.  This person will need to
contact me.

4. Authentication
=================

The BuildBot try command can use SSH for authentication, so that is what
I have chosen.  Basically, you will need to send me your *public* SSH
key; no passwords are needed/accepted.  You will be using a common user
to log in, and will not receive a PTY (i.e., will not have shell
access).  If you ever used gitolite, then you know the drill.  If not,
it's pretty simple.

I will provide a configuration file below.

5. Installing buildbot
======================

In order to use the new feature, you will have to install the "buildbot"
package.  You don't need the "slave" package!  On Debian, an "apt-get
install buildbot" should do the trick.  On Fedora, "dnf install
buildbot".  However, Fedora enables the BuildBot systemd service after
the installation; I strongly recommend you disable that, unless you
intend to run a BuildBot master on your machine.

6. The "buildbot try" command
=============================

After installing the package, the only command you need to know is
"buildbot try".  It will be responsible for generating a patch out of
the git tree you're in (assuming you don't provide a patch explicitly),
connecting to the server and submitting the patch to be tested.

I strongly recommend that you read the command's documentation at:

  <http://docs.buildbot.net/current/manual/cmdline.html#try>

However, the usage is pretty basic.  There are two modes: you can either
be in a local branch and ask "buildbot try" to automatically figure out
the patch to be tested, or you can provide the patch file explicitly.

In the first mode, "buildbot try" will generate a diff of your local
branch against your "master" branch, and will instruct the BuildBot
master to test the patch *using the master's branch HEAD revision*.
This means that if you want to generate a patch to be tested against the
latest commit of the official repository, you have to do a "git pull" on
your master branch first (and rebase your local branch, of course).

The second mode accepts a patch file via the "--diff" parameter.
Originally, "buildbot try" will not specify a revision against which the
patch needs to be tested; however, I have modified that behaviour on the
server side, and the provided patch will always be tested against the
latest commit of the official repository.

7. Configuration file for "buildbot try"
========================================

Here is the configuration file that you need to use for the "buildbot
try" command to work.  This file needs to be named
<$HOME/.buildbot/options>.

  $> cat $HOME/.buildbot/options
  # The method of connection to the BuildBot master.
  try_connect = 'ssh'

  # The host of the BuildBot master.
  try_host = 'gdb-build.sergiodj.net'

  # Our username.  We will always use this username!
  try_username = 'gdb-buildbot'

  # The jobdir on BuildBot master (i.e., the directory where "buildbot
  # try" will put the request files.
  try_jobdir = 'gdb/try_ssh_jobdir'

  # The VCS we are using for the project.
  try_vc = 'git'

  # The branch against which we will try our patches.  Currently, we
  # only support 'master'.
  try_branch = 'master'

  # Who am I?
  # NOTE: YOU NEED TO USE THE FORMAT "NAME <EMAIL>" IN THIS FIELD!
  try_who = 'John Smith <john@example.com>'

  # The builders I want to test my patch in.
  # The list of available builder can be found in:
  #  <http://sourceware.org/gdb/wiki/BuildBot#Available_Builders>
  try_builders = [ "Fedora-x86_64-m64",
                   "Fedora-x86_64-m32",
                   "Fedora-x86_64-native-gdbserver-m64",
                   "Fedora-x86_64-native-gdbserver-m32",
                   "Fedora-x86_64-native-extended-gdbserver-m64",
                   "Fedora-x86_64-native-extended-gdbserver-m32",
                   "Fedora-x86_64-cc-with-index",
                   "Fedora-i686" ]
  $>

It is good to mention that most of these options can be overriden by
command line options, so you can (for example) choose to test your patch
against only a subset of the available try_builders.

8. Notifications
================

The "buildbot try" command offers a way to wait for the builds to
complete on the command line, but this doesn't work when you're using
SSH.  However, you will receive e-mails from our BuildBot notifying you
about the results of all of your try builds.  The e-mails will be very
similar to those sent to the gdb-testers mailing list.

9. Build specifics
==================

As mentioned above, the try builds will almost always be performed
against the latest commit on the official repository.  They will also
run using the very same flags that are being used for each builder
(unfortunately, there is no way to change these flags on try builds).
And the results will be compared against the last results that our
BuildBot has on file.

For example, if our BuildBot has just tested the commit 0abcd on a
certain builder, and then you request it to test your patch on the same
builder, BuildBot will run the full build of your patch, generate the
testsuite logs, and compare them against the testsuite logs generated by
testing 0abcd.  The known XFAIL files for that specific builder will
also be taken into account, i.e., they will be discarded when comparing
the testsuite logs.

Last, but not least, your try build will generate its own testsuite
logs, which will be recorded in that builder's git repository, available
at:

  <http://gdb-build.sergiodj.net/cgit>

10. More info
=============

I guess this e-mail already covers quite a lot, but I also intend to
populate our wiki with more info.  The page will be:

  <https://sourceware.org/gdb/wiki/BuildBot#Try_Jobs>

And, of course, I am open to suggestions for improvements (can't promise
I'll be able to dedicate much time these days, though).


----

I guess that is it.  Please let me know if you have any questions, and I
will be happy to (try to) answer!

Cheers,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: [GDB BuildBot] New "Try Server"

[Antoine Tremblay]

Sergio Durigan Junior writes:

> Last, but not least, your try build will generate its own testsuite
> logs, which will be recorded in that builder's git repository, available
> at:
>
>   <http://gdb-build.sergiodj.net/cgit>

Humm won't that "pollute" the builder results ?

I mean, if the builder is testing commit 1 2 3 and that those are
commits that were done on master but there's a try patch coming in
between named say 7 and it has higher priority.

You will then have 1 2 7 3 being tested.

Then when we want to check the results of 1 2 3 won't it be confusing to
see 7 there ? Will there be an indication that it's a try patch ?

Also since the regressions are calculated from one build to the next
won't that possibly be a problem if let's say build 7 introduces a FAIL,
then build 3 has the same FAIL, but build 2 had a PASS ? We would then
miss a regression on a master commit.

Should we have separate try builders to avoid that?

I'm also curious about what happens if you send it a series of patches,
will it squash them ?

In any case thanks for working on this :) I'm sure it will be quite
useful.

Re: [GDB BuildBot] New "Try Server"

[Sergio Durigan Junior]

On Friday, July 29 2016, Antoine Tremblay wrote:

> Sergio Durigan Junior writes:
>
>> Last, but not least, your try build will generate its own testsuite
>> logs, which will be recorded in that builder's git repository, available
>> at:
>>
>>   <http://gdb-build.sergiodj.net/cgit>
>
> Humm won't that "pollute" the builder results ?
>
> I mean, if the builder is testing commit 1 2 3 and that those are
> commits that were done on master but there's a try patch coming in
> between named say 7 and it has higher priority.
>
> You will then have 1 2 7 3 being tested.
>
> Then when we want to check the results of 1 2 3 won't it be confusing to
> see 7 there ? Will there be an indication that it's a try patch ?

Yeah, that was my concern as well.  I took extra care when recording the
results on the git repo.

So, as you can see on the repositories for each builder I record the
following files:

  - baseline
  - gdb.log
  - gdb.sum
  - previous_gdb.sum

When a normal build is triggered, BuildBot will:

  - Copy the current gdb.sum to previous_gdb.sum

  - Perform the build

  - Upload the gdb.log file from the buildslave

  - Use the current gdb.sum to calculate the regressions against the new
    gdb.sum (generated by the testsuite)

  - Update the current gdb.sum with the contents of the new gdb.sum

  - Save everything on the repo

Now, when a try build is triggered, here's what will happen:

  - Perform the build

  - Upload the gdb.log file from the buildslave

  - Use the current gdb.sum to calculate the regressions against the new
    gdb.sum (generated by the testsuite)

  - Update the current trybuild_gdb.sum with the contents of the new gdb.sum

  - Save everything on the repo

So, as you can see, on a try build we don't mess with the files
necessary to calculate the regressions on a regular build.

> Also since the regressions are calculated from one build to the next
> won't that possibly be a problem if let's say build 7 introduces a FAIL,
> then build 3 has the same FAIL, but build 2 had a PASS ? We would then
> miss a regression on a master commit.

As explained above, we would still see the regression happening because
the gdb.sum file is not touched (just read) on a try build.

> Should we have separate try builders to avoid that?

My initial thought was that, but having more builders will pollute the
web interface (although there is probably a way to suppress them to be
displayed), and will only create more repositories on the buildmaster.
But if I had infinite resources, then yeah, more builders would probably
make sense :-).

> I'm also curious about what happens if you send it a series of patches,
> will it squash them ?

If you use the first method I explained (having a local branch and
invoking "buildbot try" without the "--diff" option), then it will
squash all your local commits into one patch.  If you use the second
method ("--diff" option), then IIRC you can only send one patch.

> In any case thanks for working on this :) I'm sure it will be quite
> useful.

My pleasure!

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Re: [GDB BuildBot] New "Try Server"

[Antoine Tremblay]

Sergio Durigan Junior writes:

> On Friday, July 29 2016, Antoine Tremblay wrote:
>
>> Sergio Durigan Junior writes:
>>
>>> Last, but not least, your try build will generate its own testsuite
>>> logs, which will be recorded in that builder's git repository, available
>>> at:
>>>
>>>   <http://gdb-build.sergiodj.net/cgit>
>>
>> Humm won't that "pollute" the builder results ?
>>
>> I mean, if the builder is testing commit 1 2 3 and that those are
>> commits that were done on master but there's a try patch coming in
>> between named say 7 and it has higher priority.
>>
>> You will then have 1 2 7 3 being tested.
>>
>> Then when we want to check the results of 1 2 3 won't it be confusing to
>> see 7 there ? Will there be an indication that it's a try patch ?
>
> Yeah, that was my concern as well.  I took extra care when recording the
> results on the git repo.
>
> So, as you can see on the repositories for each builder I record the
> following files:
>
>   - baseline
>   - gdb.log
>   - gdb.sum
>   - previous_gdb.sum
>
> When a normal build is triggered, BuildBot will:
>
>   - Copy the current gdb.sum to previous_gdb.sum
>
>   - Perform the build
>
>   - Upload the gdb.log file from the buildslave
>
>   - Use the current gdb.sum to calculate the regressions against the new
>     gdb.sum (generated by the testsuite)
>
>   - Update the current gdb.sum with the contents of the new gdb.sum
>
>   - Save everything on the repo
>
> Now, when a try build is triggered, here's what will happen:
>
>   - Perform the build
>
>   - Upload the gdb.log file from the buildslave
>
>   - Use the current gdb.sum to calculate the regressions against the new
>     gdb.sum (generated by the testsuite)
>
>   - Update the current trybuild_gdb.sum with the contents of the new gdb.sum
>
>   - Save everything on the repo
>
> So, as you can see, on a try build we don't mess with the files
> necessary to calculate the regressions on a regular build.
>

Ha great :) that's a good way indeed and it avoids the builders
pollution.

> If you use the first method I explained (having a local branch and
> invoking "buildbot try" without the "--diff" option), then it will
> squash all your local commits into one patch.  If you use the second
> method ("--diff" option), then IIRC you can only send one patch.
>

OK. Worst case I guess one could script it to send a series without much
effort or add it to its git rebase hook...

Thanks,
Antoine

Re: [GDB BuildBot] New "Try Server"

[Sergio Durigan Junior]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

On Thursday, July 28 2016, I wrote:

> Hi there,

Hi again,

I've now updated the wiki page with instructions on the "Try" scheduler:

  <https://sourceware.org/gdb/wiki/BuildBot#Try_Jobs>

It's basically a copy-and-paste of my first e-mail, but a few things
have changed:

1) From now on, I will provide access to our Try scheduler to anyone who
has commit access to the official GDB repository.  I've decided to lift
the initial restrictions because I think they're too strict.

2) I've also added a bunch of new builders to the list of supported
builders for the Try scheduler.  Take a look at
<https://sourceware.org/gdb/wiki/BuildBot#Available_Builders> to see the
current list.

I was also noticing a few problems with e-mail notifications for the Try
scheduler, but these should be fixed now.

Let me know if you have any questions, comments or suggestions!

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]