gitsigur for protecting git repo integrity

gitsigur for protecting git repo integrity

[Frank Ch. Eigler]

Hi -

I'd like to share a little gadget I've been working on recently.

It's a prototype git server hook for allowing participating projects
to check and/or enforce that commits to certain branches of
shared-access git repos such as those on sourceware are properly
gpg-signed.  "properly" means signed with keys that are designated as
valid for the particular committer, so as to further protect repos -
from impersonation.  Those keys are held in an auxiliary git repo.

It's a small, self-contained python3 script (plus a man page), already
deployed to gitsigur's own git repo (in enforcing mode!), and onto
elfutils & bunsen master branches in permissive mode (so it only
tests, but accepts anyway).  Having a project gradually opt-in is
trivial for a shell-capable project admin:

 - add a +x .git/hooks/update file containing something like like:

   #! /bin/sh
   set -e
   /sourceware/projects/gitsigur-home/install/bin/gitsigur "$1" "$2" "$3"

 - add a few configuration parameters to .git/config
 
   [gitsigur]
       keygitrepo = /sourceware/projects/gitsigur-home/keygitrepo
       checkref = refs/heads/master

 - and gradually send me/us participating folks' gpg public keys to
   add to the new git://sourceware.org/git/keygitrepo.git keyring


That's it!  If in enforcing mode, this is what a push attempt would
see on an unsigned commit:

remote: gitsigur checking (enforcing) against keygitrepo /git/keygitrepo.git branch keymaster
remote: ⨯ commit 3a3c69aafa549aad042ed2210bae36fa7fe0ced8 not signed
remote: gitsigur result: ⨯ failure
remote: error: hook declined to update refs/heads/master
To ssh://sourceware.org/git/gitsigur.git
 ! [remote rejected]           master -> master (hook declined)
  error: failed to push some refs to 'ssh://sourceware.org/git/gitsigur.git'
  
On the other hand, a happier outcome looks like this:

remote: gitsigur checking (enforcing) against keygitrepo /sourceware/projects/gitsigur-home/keygitrepo branch keymaster
remote: ✓ commit d4bad9409204c887e7786427caf0b1919fd3f09d signed by authorized key for fche@redhat.com
remote: gitsigur result: ✓ success
To ssh://sourceware.org/git/gitsigur.git
   9fa6f36a615f..d4bad9409204  master -> master

In the default permissive mode, even failure gets a trophy.


From the developer side, signing git commits with gpg is not too bad.
With tools like gpg-agent mediating one's credentials, one's not stuck
typing passphrases all day.

https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work


Future:

The gadget scratches my itch already.

Depending on interest etc. and time availability, one can envision
some self-serve management of the repo that stores gpg public keys,
maybe interoperation with public gpg key servers, maybe dabbling in
web-of-trust type things.  Maybe gamification so projects and people
who commit signed things, thereby contributing to a secure supply
chain, get rewarded with electronic tchotchkes.  Maybe allow
per-branch keyring configuration.  Maybe toolshed some jargon?  What
do you think?


Sauce:

https://sourceware.org/git/gitsigur.git
https://sourceware.org/git/keygitrepo.git

Similarish:

https://gitweb.gentoo.org/infra/githooks.git/tree/local/update-02-gpg
[and probably a bunch of others]


- FChE

Re: gitsigur for protecting git repo integrity

[Mark Wielaard]

Hi Frank,

CC, FoxBoron and Sam who we were discussing some of this on irc, but
who I believe are not on overseers@.

On Fri, Jun 16, 2023 at 08:03:00PM -0400, Frank Ch. Eigler via Overseers wrote:
> I'd like to share a little gadget I've been working on recently.
> 
> It's a prototype git server hook for allowing participating projects
> to check and/or enforce that commits to certain branches of
> shared-access git repos such as those on sourceware are properly
> gpg-signed.  "properly" means signed with keys that are designated as
> valid for the particular committer, so as to further protect repos -
> from impersonation.  Those keys are held in an auxiliary git repo.
> 
> It's a small, self-contained python3 script (plus a man page), already
> deployed to gitsigur's own git repo (in enforcing mode!), and onto
> elfutils & bunsen master branches in permissive mode (so it only
> tests, but accepts anyway).  Having a project gradually opt-in is
> trivial for a shell-capable project admin:
> 
>  - add a +x .git/hooks/update file containing something like like:
> 
>    #! /bin/sh
>    set -e
>    /sourceware/projects/gitsigur-home/install/bin/gitsigur "$1" "$2" "$3"
> 
>  - add a few configuration parameters to .git/config
>  
>    [gitsigur]
>        keygitrepo = /sourceware/projects/gitsigur-home/keygitrepo
>        checkref = refs/heads/master
> 
>  - and gradually send me/us participating folks' gpg public keys to
>    add to the new git://sourceware.org/git/keygitrepo.git keyring
> 
> 
> That's it!  If in enforcing mode, this is what a push attempt would
> see on an unsigned commit:
> 
> remote: gitsigur checking (enforcing) against keygitrepo /git/keygitrepo.git branch keymaster
> remote: ⨯ commit 3a3c69aafa549aad042ed2210bae36fa7fe0ced8 not signed
> remote: gitsigur result: ⨯ failure
> remote: error: hook declined to update refs/heads/master
> To ssh://sourceware.org/git/gitsigur.git
>  ! [remote rejected]           master -> master (hook declined)
>   error: failed to push some refs to 'ssh://sourceware.org/git/gitsigur.git'
>   
> On the other hand, a happier outcome looks like this:
> 
> remote: gitsigur checking (enforcing) against keygitrepo /sourceware/projects/gitsigur-home/keygitrepo branch keymaster
> remote: ✓ commit d4bad9409204c887e7786427caf0b1919fd3f09d signed by authorized key for fche@redhat.com
> remote: gitsigur result: ✓ success
> To ssh://sourceware.org/git/gitsigur.git
>    9fa6f36a615f..d4bad9409204  master -> master
> 
> In the default permissive mode, even failure gets a trophy.
> 
> 
> From the developer side, signing git commits with gpg is not too bad.
> With tools like gpg-agent mediating one's credentials, one's not stuck
> typing passphrases all day.
> 
> https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
> 
> 
> Future:
> 
> The gadget scratches my itch already.
> 
> Depending on interest etc. and time availability, one can envision
> some self-serve management of the repo that stores gpg public keys,
> maybe interoperation with public gpg key servers, maybe dabbling in
> web-of-trust type things.  Maybe gamification so projects and people
> who commit signed things, thereby contributing to a secure supply
> chain, get rewarded with electronic tchotchkes.  Maybe allow
> per-branch keyring configuration.  Maybe toolshed some jargon?  What
> do you think?

I like the general idea of (optionally) signing commits. And having an
associated store of known keys.

As long as there are also unsigned or unknown signed commits it makes
sense to also introduce some kind of transparancy log so people can
check commits came in through a (ssh authenticated) receive-pack (and
were not to result of direct manipulation of a repo on the server).

I don't think enforcing mode will be very popular on normal
development branches. But I can see it being something you might want
for release branches. So a configuration where only release
maintainers may create and push to release branches and you can check
that all commits are signed up to the branch point.

Also you might want to have a mode between permissive and enforcing,
that simply checks all commits are signed? But maybe not having the
public key for a signed commit doesn't really make sense?

Does it make sense for there to be a mode that requires (just) tags to
be signed?

> Sauce:
> 
> https://sourceware.org/git/gitsigur.git
> https://sourceware.org/git/keygitrepo.git
> 
> Similarish:
> 
> https://gitweb.gentoo.org/infra/githooks.git/tree/local/update-02-gpg
> [and probably a bunch of others]
> 
> 
> - FChE

Re: gitsigur for protecting git repo integrity

[Frank Ch. Eigler]

Hi -

> I like the general idea of (optionally) signing commits. And having an
> associated store of known keys.

Righto.

> As long as there are also unsigned or unknown signed commits it makes
> sense to also introduce some kind of transparancy log so people can
> check commits came in through a (ssh authenticated) receive-pack (and
> were not to result of direct manipulation of a repo on the server).

Manipulation on the server could not result in creation or editing of
signed commits, since the server (by design) does not hold any crypto
credentials.  That's one of the benefits of habitually signing git
content.

> I don't think enforcing mode will be very popular on normal
> development branches. But I can see it being something you might want
> for release branches. [...]

Who knows, maybe.  Given that releases come from development branches,
and given that signing your commits is rather lightweight, eventually
enough people could get used to it and to the assurances to just
toggle the switch for important branches.

But the script is configurable.  Any project can:

- direct their sigur hook to a key repo of their choice, or a shared one

- have multiple active sigur hooks, configured differently, so as to
  enforce distinct policies for different branches


> [...]
> Does it make sense for there to be a mode that requires (just) tags to
> be signed?

I suspect that wouldn't need to be a mode, just another configuration

     sigur --checkref='ref/tags/*' --mode=enforcing [...]


- FChE

Re: gitsigur for protecting git repo integrity

[Frank Ch. Eigler]

Hi -

> I'd like to share a little gadget I've been working on recently.
> 
> It's a prototype git server hook for allowing participating projects
> to check and/or enforce that commits to certain branches of
> shared-access git repos such as those on sourceware are properly
> gpg-signed.  [...]

Some questions have arisen offline about the relationship of this
script to systems such as sigstore and b4.  Here's a rough comparison
of the three.  This is pretty terse point-by-point analysis.  I'd be
happy to elaborate on any aspect of it.

tl;dr: Overall, it turns out to be more of a composition situation
rather than competition.


Sources: [lines-of-source language]

* gitsigur:  https://sourceware.org/git/gitsigur.git  [200 python]

* sigstore:  https://github.com/sigstore/gitsign, /rekor, /fulcio [73000 go]

* b4:        https://github.com/mricon/b4 [8500 python]


Scope:

* gitsigur:  enforcement of gpg-signed git commits at git server;
             keyring storage in some git repo

* sigstore:  developer-side git signature tooling, via custom
             temporary x509 certificates bound to a web openid identity

* b4:        signed/secure git commit text transport over email


Scope overlap:

* gitsigur/b4: none, though the signed git commit message can include the
               public-inbox url for authenticating the chain of custody

* sigstore/b4: ditto

* gitsigur/sigstore: use of gpg vs custom signatures on git commits
                     (nb: sigstore upstream development uses github's
                     server-side gpg signatures, does not eat sigstore
                     dog-food itself)

* gitsigur/gitlab: signing same, verification analogous 
  https://docs.gitlab.com/ee/user/project/repository/gpg_signed_commits/index.html
  https://docs.gitlab.com/ee/user/project/repository/push_rules.html

* gitsigur/github: signing same, verification analogous
  https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key
  https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches


Infrastructure required:

* gitsigur:  a git server

* sigstore:  an openid web server, a certificate generator web server,
             audit trail web server, timestamp server

* b4:        a public-inbox email server


Developer workflow - pushing code:

* gitsigur:  create & share gpg key once, use builtin git gpg support
             to sign locally; normal git push to pass gitsigur
             enforcing/permissive gate hook

* sigstore:  install "gitsign" as replacement git signing/crypto
             engine; use openid web system to authenticate against
             some identity server; public certificate generator system
             creates temp certificate; custom git commit signature
             scheme includes entire temp certificate abeam the
             signature itself; git commit news may be broadcast to
             audit trail; normal git push, no server-side gating hook

* b4:        developer posts git patches with b4 wrapper to add gpg
             signature; maintainer uses b4 wrapper to check integrity of
             emailed patch series; normal git commit & push


Developer workflow - verifying commits from git repo:

* gitsigur:  download keyring git repo; run gitsigur locally against
             range of commits

* sigstore:  run gitsign verify on local repo, with the certificate
             keyring stored right within the commits; also possibly
             connecting with public audit trail servers

* b4:        not applicable


Hosting git server workflow:

* gitsigur:  one git update hook

* sigstore:  not applicable as server-side gating does not appear to be part of sigstore

* b4:        not applicable


Composition possibilities:

* gitsigur/b4:  use b4 to send/receive patches by email, gpg-sign resulting
                commit and enforce at server side via gitsigur

* gitsigur/sigstore:  use sigstore to sign commits; teach gitsigur to detect
                      sigstore flavoured signatures and delegate verification
                      to sigstore gitsign, then enforce policy similarly 

* gitsigur/sigstore 2: teach gitsigur to announce receipt of verified commits
                       to sigstore (rekor) public audit-trail

* gitsigur/ssh/x509:  extend gitsigur to support ssh and x509 standard
                      git signature styles, let developers choose
                      their favorite; store keys/certs in same
                      keygitrepo


- FChE

Re: gitsigur for protecting git repo integrity

[Mark Wielaard]

Hi Frank,

On Thu, Jun 29, 2023 at 02:55:06PM -0400, Frank Ch. Eigler via Overseers wrote:
> > I'd like to share a little gadget I've been working on recently.
> > 
> > It's a prototype git server hook for allowing participating projects
> > to check and/or enforce that commits to certain branches of
> > shared-access git repos such as those on sourceware are properly
> > gpg-signed.  [...]
> 
> Some questions have arisen offline about the relationship of this
> script to systems such as sigstore and b4.  Here's a rough comparison
> of the three.  This is pretty terse point-by-point analysis.  I'd be
> happy to elaborate on any aspect of it.
> 
> tl;dr: Overall, it turns out to be more of a composition situation
> rather than competition.

Very nice, this was very helpful.

On irc we also discussed signed git pushes (which are now also enabled
on sourceware). Which provides another "layer" of integrity.

As reply to our fosstodon post
https://fosstodon.org/@sourceware/110629743586988452 someone from the
tor project posted an overview of "Git repository integrity solutions"
they wrote up for their project:
https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#git-repository-integrity-solutions

It references this discussion, so now I am completing the circle :)

Cheers,

Mark

Re: gitsigur for protecting git repo integrity

[Mark Wielaard]

Hi all,

Collecting more fediverse replies.

On Tue, Jul 04, 2023 at 10:32:45AM +0200, Mark Wielaard via Overseers wrote:
> On Thu, Jun 29, 2023 at 02:55:06PM -0400, Frank Ch. Eigler via Overseers wrote:
> > > I'd like to share a little gadget I've been working on recently.
> > > 
> > > It's a prototype git server hook for allowing participating projects
> > > to check and/or enforce that commits to certain branches of
> > > shared-access git repos such as those on sourceware are properly
> > > gpg-signed.  [...]
> > 
> > Some questions have arisen offline about the relationship of this
> > script to systems such as sigstore and b4.  Here's a rough comparison
> > of the three.  This is pretty terse point-by-point analysis.  I'd be
> > happy to elaborate on any aspect of it.
> > 
> > tl;dr: Overall, it turns out to be more of a composition situation
> > rather than competition.
> 
> Very nice, this was very helpful.
> 
> On irc we also discussed signed git pushes (which are now also enabled
> on sourceware). Which provides another "layer" of integrity.
> 
> As reply to our fosstodon post
> https://fosstodon.org/@sourceware/110629743586988452 someone from the
> tor project posted an overview of "Git repository integrity solutions"
> they wrote up for their project:
> https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/gitlab#git-repository-integrity-solutions
> 
> It references this discussion, so now I am completing the circle :)

Another reply (CCing the author):
https://toot.aquilenet.fr/@civodul/110658002235937665

  @sourceware To authenticate source code, the mechanism initially
  developed for Guix:

  https://doi.org/10.22152/programming-journal.org/2023/7/1
  https://guix.gnu.org/manual/en/html_node/Invoking-guix-git-authenticate.html

  My understanding is that gitsigur checks signatures against an
  out-of-band list of authorized keys, which isn’t very useful because
  the set of authorized committers changes over time.

Re: gitsigur for protecting git repo integrity

[Frank Ch. Eigler]

Hi -

>   My understanding is that gitsigur checks signatures against an
>   out-of-band list of authorized keys, which isn’t very useful because
>   the set of authorized committers changes over time.

The list of authorized keys is stored in a selected branch of a
selected git repo, whose contents may change over time (and: maintain
an audit trail) (and: may itself be protected by security mechanisms
such as gitsigur).

- FChE

Re: gitsigur for protecting git repo integrity

[Ludovic Courtès]

Hi,

"Frank Ch. Eigler" <fche@redhat.com> skribis:

>>   My understanding is that gitsigur checks signatures against an
>>   out-of-band list of authorized keys, which isn’t very useful because
>>   the set of authorized committers changes over time.
>
> The list of authorized keys is stored in a selected branch

If it’s in another branch than the code it’s about, how can you tell
whether a key was authorized at a given point in the commit history?

Thanks,
Ludo’.

Re: gitsigur for protecting git repo integrity

[Frank Ch. Eigler]

Hi -

> >>   My understanding is that gitsigur checks signatures against an
> >>   out-of-band list of authorized keys, which isn’t very useful because
> >>   the set of authorized committers changes over time.
> >
> > The list of authorized keys is stored in a selected branch
> 
> If it’s in another branch than the code it’s about, how can you tell
> whether a key was authorized at a given point in the commit history?

You're talking about retrospectively verifying old commit signatures,
rather than verifying eligilibity at the time of a push.  gitsigur at
uses the "current" contents of the keymaster repo/branch for the list
of public keys.  It could also look back in time, relying on that
repo's commit timeline, to inspect the time-varying mapping.  There is
enough information there, so it's a SMOP.

- FChE

Re: gitsigur for protecting git repo integrity

[Ludovic Courtès]

Hi Frank and all,

"Frank Ch. Eigler" <fche@elastic.org> skribis:

>> >>   My understanding is that gitsigur checks signatures against an
>> >>   out-of-band list of authorized keys, which isn’t very useful because
>> >>   the set of authorized committers changes over time.
>> >
>> > The list of authorized keys is stored in a selected branch
>> 
>> If it’s in another branch than the code it’s about, how can you tell
>> whether a key was authorized at a given point in the commit history?
>
> You're talking about retrospectively verifying old commit signatures,
> rather than verifying eligilibity at the time of a push.

Both.  Assume a contributor has a genuine checkout; how do you ensure
that when they eventually run ‘git pull’ they can authenticate their
updated checkout?  You need to somehow convey the updated set of
authorized committers to everyone who pulls from the repo.

This is what ‘guix git authenticate’ addresses.

> gitsigur at uses the "current" contents of the keymaster repo/branch
> for the list of public keys.  It could also look back in time, relying
> on that repo's commit timeline, to inspect the time-varying mapping.
> There is enough information there, so it's a SMOP.

“SMOP”?  I think info about the set of authorized keys should be stored
in-band, in the repo.  If you maintain it out-of-band, then you can try
to match timelines as you write, but it’s just an approximation, it’s
unreliable (you cannot rely on timestamps in Git commits, for instance),
and it doesn’t work once you have multiple branches.

I think you should take a look at Sections 4 and 5 of
<https://arxiv.org/pdf/2206.14606v1.pdf>.  Maybe you’ll decide that
‘guix git authenticate’ is unsuitable and maybe you’ll end up extending
gitsigur instead, but I think the discussion there is worth a look
because it’s about precisely what we’re discussing.

HTH!

Ludo’.

Re: gitsigur for protecting git repo integrity

[Frank Ch. Eigler]

Hi -

> > You're talking about retrospectively verifying old commit signatures,
> > rather than verifying eligilibity at the time of a push.
> 
> Both.

OK but you're talking exclusively about the former:

> Assume a contributor has a genuine checkout; how do you ensure
> that when they eventually run ‘git pull’ they can authenticate their
> updated checkout?  You need to somehow convey the updated set of
> authorized committers to everyone who pulls from the repo.

One can pull the git repo containing the authorized pubkeys.  (At this
point, this would be manual; gitsigur should learn to do a git clone
if given remote URL.)

> This is what ‘guix git authenticate’ addresses.

OK.

> > gitsigur at uses the "current" contents of the keymaster repo/branch
> > for the list of public keys.  It could also look back in time, relying
> > on that repo's commit timeline, to inspect the time-varying mapping.
> > There is enough information there, so it's a SMOP.
> 
> “SMOP”?

("small matter of programming")

> I think info about the set of authorized keys should be stored
> in-band, in the repo.  If you maintain it out-of-band, then you can
> try to match timelines as you write, but it’s just an approximation,
> it’s unreliable (you cannot rely on timestamps in Git commits, for
> instance),

If the pubkeys are in the same repo but in another branch, the
timestamp unreliability problem still exists, such as it is.
(gitsigur can be configured for this layout already.)

If the pubkeys are stored within the same branch as the payload source
files, then this requires a change to the project source management.
Even this configuration would work with gitsigur, just a bit ugly.

> and it doesn’t work once you have multiple branches.

I don't understand this concern.


> I think you should take a look at Sections 4 and 5 of
> <https://arxiv.org/pdf/2206.14606v1.pdf>.  [...]

Will read, thanks.

- FChE