patchwork upgrade week

patchwork upgrade week

[Siddhesh Poyarekar]

Hello,

This week I'm going to attempt to do some updates to the patchwork 
infrastructure to get it up to the latest version.  You'll likely see 
tiny downtimes or if I mess up, large periods of it :D  A bulk of the 
manual patchwork usage happens on Mondays during the review call, so I'm 
going to start after the call.

If you see patches that have gone missing off patchwork or are having 
any trouble with your account, please let me know.

Thanks,
Siddhesh

Re: patchwork upgrade week

[Siddhesh Poyarekar]

On 20/06/2022 15:56, Siddhesh Poyarekar wrote:
> Hello,
> 
> This week I'm going to attempt to do some updates to the patchwork 
> infrastructure to get it up to the latest version.  You'll likely see 
> tiny downtimes or if I mess up, large periods of it :D  A bulk of the 
> manual patchwork usage happens on Mondays during the review call, so I'm 
> going to start after the call.
> 
> If you see patches that have gone missing off patchwork or are having 
> any trouble with your account, please let me know.

This is now done.  We're at patchwork 3.0.5, which is the latest 
available.  Our django installation is at 3.1.14 because 3.1 is the 
latest patchwork officially claims to support at the moment.

I did a quick test with django 3.2.13 and the server did come up but 
with some warnings.  Hopefully once patchwork is updated to support this 
and future updates will be easy enough.

I have changed things around a bit in the sourceware patchwork directory 
and have documented steps for testing and deploying upgrades to 
patchwork, django and anything else in a README file in the home directory.

Cheers,
Sid

Re: patchwork upgrade week

[Mark Wielaard]

Hi Siddhesh,

On Wed, Jun 22, 2022 at 12:23:56PM +0530, Siddhesh Poyarekar wrote:
> On 20/06/2022 15:56, Siddhesh Poyarekar wrote:
> This is now done.  We're at patchwork 3.0.5, which is the latest available.
> Our django installation is at 3.1.14 because 3.1 is the latest patchwork
> officially claims to support at the moment.
> [...]
> I have changed things around a bit in the sourceware patchwork directory and
> have documented steps for testing and deploying upgrades to patchwork,
> django and anything else in a README file in the home directory.

Thanks so much for doing this. And thanks for the extensive
README. For others who wish to help with upgrades in the future,
please apply for the patchwork group.

We were working on adding glibc to builder.sourceware.org (which was
how I notice the build breakage I just reported). And you might have
seen that for binutils and gdb we are experimenting with git users try
branches: https://sourceware.org/pipermail/gdb/2022-June/050173.html

Once the new hardware is installed we could also enable that for
glibc, but that would only be for people who already have commit
access. And you already have the CICD trybot integrated with patchwork
so any patch submitter can see try build results.

So I was hoping to integrate the buildbot with the patchwork based
trybot. But it would be good to have some way to authenticate the
patch as genuine before throwing it at the buildbot-worker. I was
hoping that could be done by a project admin setting the state of the
patch to some "please-try" value. But it looks like a (rogue) user can
set the state on their own patch. So relying on the patchwork patch
state seems not secure.

Or is there a state (maybe a check state?) that we can make sure can
only be set by project admins?

If not, how else can we authenticate a patch as "OK to let the
buildbot do a try build?"

Cheers,

Mark

Re: patchwork upgrade week

[Siddhesh Poyarekar]

On 23/06/2022 02:57, Mark Wielaard wrote:
> So I was hoping to integrate the buildbot with the patchwork based
> trybot. But it would be good to have some way to authenticate the
> patch as genuine before throwing it at the buildbot-worker. I was
> hoping that could be done by a project admin setting the state of the
> patch to some "please-try" value. But it looks like a (rogue) user can
> set the state on their own patch. So relying on the patchwork patch
> state seems not secure.
> 
> Or is there a state (maybe a check state?) that we can make sure can
> only be set by project admins?
> 
> If not, how else can we authenticate a patch as "OK to let the
> buildbot do a try build?"

DJ is working on an email based workflow.  Another method (that Frank 
proposed on IRC recently) could be for a trybot to track namespace 
branches for all committers on sourceware and trigger builds on specific 
names, e.g. siddhesh/buildbot/*.  That way maintainers can manually 
trigger builds by pushing their changes to that specific branch namespace.

Sid

Re: patchwork upgrade week

[Mark Wielaard]

Hi Siddhesh,

On Thu, 2022-06-23 at 07:46 +0530, Siddhesh Poyarekar wrote:
> On 23/06/2022 02:57, Mark Wielaard wrote:
> > So I was hoping to integrate the buildbot with the patchwork based
> > trybot. But it would be good to have some way to authenticate the
> > patch as genuine before throwing it at the buildbot-worker. I was
> > hoping that could be done by a project admin setting the state of the
> > patch to some "please-try" value. But it looks like a (rogue) user can
> > set the state on their own patch. So relying on the patchwork patch
> > state seems not secure.
> > 
> > Or is there a state (maybe a check state?) that we can make sure can
> > only be set by project admins?
> > 
> > If not, how else can we authenticate a patch as "OK to let the
> > buildbot do a try build?"
> 
> DJ is working on an email based workflow.  Another method (that Frank 
> proposed on IRC recently) could be for a trybot to track namespace 
> branches for all committers on sourceware and trigger builds on specific 
> names, e.g. siddhesh/buildbot/*.  That way maintainers can manually 
> trigger builds by pushing their changes to that specific branch namespace.

OK, so that would work like the git try branches, but tells the
buildbot to also find the same patch on patchwork to add a check
result?

Could we then use patchwork/hasher.py to calculate the hash of the
commit and look it up in the project patch list so it can be update it
with the builder check result?

That sounds like a promising idea.

Thanks,

Mark

Re: patchwork upgrade week

[Siddhesh Poyarekar]

On 23/06/2022 13:25, Mark Wielaard wrote:
> OK, so that would work like the git try branches, but tells the
> buildbot to also find the same patch on patchwork to add a check
> result?
> 
> Could we then use patchwork/hasher.py to calculate the hash of the
> commit and look it up in the project patch list so it can be update it
> with the builder check result?

Yes, as long as the diff is identical, hasher.py should be able to find it.

Sid

Re: patchwork upgrade week

[DJ Delorie]

Mark Wielaard <mjw@gnu.org> writes:
> If not, how else can we authenticate a patch as "OK to let the
> buildbot do a try build?"

[/me is back from vacation]

The method we're using to add this type of authentication is GPG-signed
emails.  I.e. an authorized user would reply to the patch email, in the
reply give a command to their runners, and sign the command.  The system
would authenticate the signature and authorize the runners to act as
instructed.  Example:

In-Reply-To:...
...
Joe User wrote:
>
>  The patch
>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

%cicd buildbot expensive-hosts . . .
-----BEGIN PGP SIGNATURE-----
. . .
-----END PGP SIGNATURE-----


Each runner decides whose commands they honor, and which, and how.  The
curator manages the authentication, so only one keyring is needed.

Note that our existing 32-bit trybot does not use this; it assumes
patches may be malicious and plans accordingly.  This allows it to
proactively test all posted patches.

Re: patchwork upgrade week

[Mark Wielaard]

Hi DJ,

On Mon, 2022-06-27 at 13:23 -0400, DJ Delorie wrote:
> Mark Wielaard <mjw@gnu.org> writes:
> > If not, how else can we authenticate a patch as "OK to let the
> > buildbot do a try build?"
> 
> The method we're using to add this type of authentication is GPG-
> signed
> emails.  I.e. an authorized user would reply to the patch email, in the
> reply give a command to their runners, and sign the command.  The system
> would authenticate the signature and authorize the runners to act as
> instructed.  Example:
> [...]
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> %cicd buildbot expensive-hosts . . .
> -----BEGIN PGP SIGNATURE-----
> . . .
> -----END PGP SIGNATURE-----
> 
> Each runner decides whose commands they honor, and which, and how.  The
> curator manages the authentication, so only one keyring is needed.

OK, so in this case the runner would be the buildbot builders and they
would simply trust that the curator did the authentication. Is the
authentication status part of the event that the trybot receives?

Thanks,

Mark

Re: patchwork upgrade week

[DJ Delorie]

Mark Wielaard <mjw@gnu.org> writes:
> OK, so in this case the runner would be the buildbot builders and they
> would simply trust that the curator did the authentication.

The runners trust the curator, and the trybots trust the runner.

> Is the authentication status part of the event that the trybot
> receives?

I haven't gotten to that point yet, but likely yes.  It's easy to
include extra information in JSON.

I don't know off the top of my head how much information will be
provided by gpg though.