Sourceware @ Conservancy - Year One

Sourceware @ Conservancy - Year One

[Mark Wielaard]

Sourceware joined Conservancy as member project on May 15 2023.
https://sfconservancy.org/news/2023/may/15/sourceware-joins-sfc/

It was a busy year and we would like to give an overview of various
topics.

- Communications
- New and updated services
- Security
- New and upgraded hardware
- Finances
- Next year plans
- Conclusion

= Communications

In the last year we organized 12 Open Office meetings on IRC.

And posted Sourceware infrastructure community quarterly updates for
23Q2 https://inbox.sourceware.org/20230605090950.GI16634@gnu.wildebeest.org
23Q3 https://inbox.sourceware.org/20230830081253.GB26251@gnu.wildebeest.org
23Q4 https://inbox.sourceware.org/20231128101132.GE4214@gnu.wildebeest.org
24Q1 https://inbox.sourceware.org/20240227091935.GK17722@gnu.wildebeest.org

We also published the Sourceware 25 Roadmap. Preparing Sourceware for
the next 25 years. https://sourceware.org/sourceware-25-roadmap.html

Various members of the Sourceware Project Leadership Committee and
Conservancy staff attended the GNU Tools Cauldron in 2023 and FOSDEM
in 2024 to meet in person.

The Software Freedom Conservancy extended the use of their Big Blue
Button instance https://bbb.sfconservancy.org/ to Sourceware projects
that want to host video meetings.

And Sourceware joined the fediverse at @sourceware@fosstodon.org
https://fosstodon.org/@sourceware

= New and updated services

https://snapshots.sourceware.org/

Thanks to OSUOSL we now have a snapshots server to publish static
artifacts from current git repos created in isolated containers.
It can be used as alternative to git hooks or cron jobs to generate
snapshots for things like:

glibc code and manual snapshots:
  https://snapshots.sourceware.org/glibc/trunk/latest/
GNU poke code and doc snapshots:
  https://snapshots.sourceware.org/gnupoke/trunk/latest/
elfutils code coverage:
  https://snapshots.sourceware.org/elfutils/coverage/latest/
libabigail website, manuals and api docs:
  https://snapshots.sourceware.org/libabigail/html-doc/latest/
Valgrind snapshots and manuals:
  https://snapshots.sourceware.org/valgrind/trunk/latest/
DWARF draft spec:
  https://snapshots.sourceware.org/dwarfstd/dwarf-spec/latest/
GDB code snapshots:
  https://snapshots.sourceware.org/gdb/trunk/latest/src/
Binutils code snapshots:
  https://snapshots.sourceware.org/binutils/trunk/latest/src/

The container files and build steps are defined through the builder
project.

The Software Heritage project https://www.softwareheritage.org/
started archiving the active git repos and the (historic) subversion
and cvs archives. This is in addition to the mirrors at SourceHut
https://sr.ht/~sourceware/

Email. No more From rewriting for patches mailinglists.
Sourceware mailinglists used From rewriting. No more! We upgraded
mailman, gave up subject prefixes, mail footers, html stripping and
reply-to mangling.

This includes the libc-alpha and gcc-patches mailinglists. The gcc
patches lists for libstdc++, libgccjit, fortran and gcc-rust. And the
lists for projects that use patchwork, newlib, elfutils, libabigail
and gdb.

Thanks to the FSF tech-team for walking us through their setup for
lists.gnu.org

https://inbox.sourceware.org/ now also "handles" HTML emails (by
stripping the HTML part) and was reindexed to include any missing
(HTML) emails.

Various projects were still creating their project homepages from
CVS. We upgraded both glibc and binutils to have a public git htdocs
repository now to which the whole community can contribute.

https://sourceware.org/cgit/binutils-htdocs/
https://sourceware.org/cgit/glibc-htdocs/

And a special thanks to ARM who have been using
https://patchwork.sourceware.org/ to provide a pre-commit testing
service for various projects.

= Security

Sourceware introduced gitsigur for protecting git repo integrity. With
comparisons, developer workflow examples and composition possibilities
for gitsigur, b4 and sigstore.
https://inbox.sourceware.org/ZJ3Tihvu6GbOb8%2FR@elastic.org/

Sourceware now also allows signed git pushes
(in addition to signed git commits).

The Common Vulnerabilities and Exposures (CVE) system seems broken and
has been issuing more and more questionable advisories. Various hosted
projects have been writing security policies to help users know which
bugs might have security implications.

https://sourceware.org/cgit/elfutils/tree/SECURITY
https://sourceware.org/cgit/binutils-gdb/tree/binutils/SECURITY.txt
https://gcc.gnu.org/cgit/gcc/tree/SECURITY.txt

The glibc project even setup their own security mailing list and CNA
(CVE Numbering Authority) publishing their own advisories:
https://sourceware.org/glibc/security.html
https://sourceware.org/cgit/glibc/tree/advisories

To double check that generated files in source repositories are really
what was intended the container builders now have an autotools
generated files checker, autoregen, for gcc, binutils and gdb:
https://inbox.sourceware.org/20231115194803.GW31613@gnu.wildebeest.org/

Sourceware hosts were not affected by the xz-backdoor. But we did
reset the https://builder.sourceware.org containers of debian-testing,
fedora-rawhide and opensuse-tumbleweed. These containers however
didn't have ssh installed, were running on isolated VMs on separate
machines from our main hosts, snapshots and backup servers.

We introduced an "aging inactive users" policy. Accounts are now
automatically disabled when not used for a year (after a warning).
https://inbox.sourceware.org/overseers/ZhCho2hjRACDztxy@elastic.org

= New and upgraded hardware

There have been complaints about overloaded builders on
https://builder.sourceware.org. So OSUOSL have provided us with
another arm64 and x86_64 server. The new servers do the larger gcc and
glibc builds so the other builders can do quicker (smaller) CI builds
without having to wait on the big jobs.

StarFive has donated 4 VisionFive-2 RISC-V boards with 8GB, 4-core
JH7110 supporting the RV64GC ISA for https://builder.sourceware.org/
Which has allowed us to setup CI (and try) builders for various
projects: annobin, binutils(+try), bzip2, debugedit, dwz,
elfutils(+try), glibc, gdb, poke, and libabigail(+try).

One of the drives in server2 broke down. It was part of a 10 drive
raid6 setup, which can take 2 bad disks before full failure. We also
have a full mirror on server3, which has a similar raid6 setup. We
ordered 3 new disks, one as replacement for the bad disk and a spare
for server2 and server3 in case of future drive failures. The drive
has been replaced and everything is running smoothly again.

Thanks to Red Hat server2 got a RAM upgrade to 512G.

= Finances

To create a hardware replacement fund we setup
https://sourceware.org/donate.html

There were $5.500+ in individual donations in the last year.

And Valgrind was picked for a FUTO https://futo.org Microgrant, which
has been donated to Sourceware through the Software Freedom
Conservancy for maintaining and expanding the infrastructure for
Valgrind and other core toolchain and developer tool projects.
FUTO then doubled their contribution to $2.000.

Thanks to our hardware and services partners we didn't have much
direct expenses. We spend ~$300 on the replacement disks and $20 on
domain registration.

Total income was $7,611.73, total expenses were $321.76.
Note that income is after currency conversions and administration costs.

Which leaves us with $7,289.97 for our current hardware replacement fund.

= Next year plans

To prepare for next year we held various open office and public email
discussions with the community and made plans for Sourceware and the
hosted projects secure software development frameworks.

https://inbox.sourceware.org/20240325100226.GL5673@gnu.wildebeest.org
https://inbox.sourceware.org/20240401150617.GF19478@gnu.wildebeest.org
https://inbox.sourceware.org/20240417232725.GC25080@gnu.wildebeest.org

After the xz-backdoor incident obviously a lot of discussions focused
on various security aspects. The Sourceware Project Leadership
Committee turned those ideas into concrete plans for next year:

Secure Sourceware Project Goals
https://sourceware.org/sourceware-security-vision.html Secure

More isolation of existing services. Modernizing account
processes. Release upload process improvements. Hardware keys for
administrators, release managers and developers. Pull-request
server. Part time junior system administrator.

We are currently working with the Conservancy to fund these plans.

= Conclusion

This first year as a Conservancy Member Project has been really good
for Sourceware and we hope to continue the relationship for many years
to come. We urge the community to support the Software Freedom
Conservancy by becoming a Conservancy Sustainer
https://sfconservancy.org/sustainer

Re: Sourceware @ Conservancy - Year One

[Maxim Kuvyrkov]

> On May 29, 2024, at 23:02, Mark Wielaard <mark@klomp.org> wrote:
> 
> Sourceware joined Conservancy as member project on May 15 2023.
> https://sfconservancy.org/news/2023/may/15/sourceware-joins-sfc/
> 
> It was a busy year and we would like to give an overview of various
> topics.
> 
> - Communications
> - New and updated services
> - Security
> - New and upgraded hardware
> - Finances
> - Next year plans
> - Conclusion
> 
> = Communications
> 
> In the last year we organized 12 Open Office meetings on IRC.
> 
> And posted Sourceware infrastructure community quarterly updates for
> 23Q2 https://inbox.sourceware.org/20230605090950.GI16634@gnu.wildebeest.org
> 23Q3 https://inbox.sourceware.org/20230830081253.GB26251@gnu.wildebeest.org
> 23Q4 https://inbox.sourceware.org/20231128101132.GE4214@gnu.wildebeest.org
> 24Q1 https://inbox.sourceware.org/20240227091935.GK17722@gnu.wildebeest.org
> 
> We also published the Sourceware 25 Roadmap. Preparing Sourceware for
> the next 25 years. https://sourceware.org/sourceware-25-roadmap.html
> 
> Various members of the Sourceware Project Leadership Committee and
> Conservancy staff attended the GNU Tools Cauldron in 2023 and FOSDEM
> in 2024 to meet in person.
> 
> The Software Freedom Conservancy extended the use of their Big Blue
> Button instance https://bbb.sfconservancy.org/ to Sourceware projects
> that want to host video meetings.
> 
> And Sourceware joined the fediverse at @sourceware@fosstodon.org
> https://fosstodon.org/@sourceware
> 
> = New and updated services
> 
> https://snapshots.sourceware.org/
> 
> Thanks to OSUOSL we now have a snapshots server to publish static
> artifacts from current git repos created in isolated containers.
> It can be used as alternative to git hooks or cron jobs to generate
> snapshots for things like:
> 
> glibc code and manual snapshots:
>  https://snapshots.sourceware.org/glibc/trunk/latest/
> GNU poke code and doc snapshots:
>  https://snapshots.sourceware.org/gnupoke/trunk/latest/
> elfutils code coverage:
>  https://snapshots.sourceware.org/elfutils/coverage/latest/
> libabigail website, manuals and api docs:
>  https://snapshots.sourceware.org/libabigail/html-doc/latest/
> Valgrind snapshots and manuals:
>  https://snapshots.sourceware.org/valgrind/trunk/latest/
> DWARF draft spec:
>  https://snapshots.sourceware.org/dwarfstd/dwarf-spec/latest/
> GDB code snapshots:
>  https://snapshots.sourceware.org/gdb/trunk/latest/src/
> Binutils code snapshots:
>  https://snapshots.sourceware.org/binutils/trunk/latest/src/
> 
> The container files and build steps are defined through the builder
> project.
> 
> The Software Heritage project https://www.softwareheritage.org/
> started archiving the active git repos and the (historic) subversion
> and cvs archives. This is in addition to the mirrors at SourceHut
> https://sr.ht/~sourceware/
> 
> Email. No more From rewriting for patches mailinglists.
> Sourceware mailinglists used From rewriting. No more! We upgraded
> mailman, gave up subject prefixes, mail footers, html stripping and
> reply-to mangling.
> 
> This includes the libc-alpha and gcc-patches mailinglists. The gcc
> patches lists for libstdc++, libgccjit, fortran and gcc-rust. And the
> lists for projects that use patchwork, newlib, elfutils, libabigail
> and gdb.
> 
> Thanks to the FSF tech-team for walking us through their setup for
> lists.gnu.org
> 
> https://inbox.sourceware.org/ now also "handles" HTML emails (by
> stripping the HTML part) and was reindexed to include any missing
> (HTML) emails.
> 
> Various projects were still creating their project homepages from
> CVS. We upgraded both glibc and binutils to have a public git htdocs
> repository now to which the whole community can contribute.
> 
> https://sourceware.org/cgit/binutils-htdocs/
> https://sourceware.org/cgit/glibc-htdocs/
> 
> And a special thanks to ARM who have been using
> https://patchwork.sourceware.org/ to provide a pre-commit testing
> service for various projects.

Hi Mark,

Thanks for the great update!

Minor nitpick: pre-commit testing for AArch64 and AArch32 architectures is provided by Linaro Toolchain Working Group (Linaro TCWG).

--
Maxim Kuvyrkov
https://www.linaro.org

> 
> = Security
> 
> Sourceware introduced gitsigur for protecting git repo integrity. With
> comparisons, developer workflow examples and composition possibilities
> for gitsigur, b4 and sigstore.
> https://inbox.sourceware.org/ZJ3Tihvu6GbOb8%2FR@elastic.org/
> 
> Sourceware now also allows signed git pushes
> (in addition to signed git commits).
> 
> The Common Vulnerabilities and Exposures (CVE) system seems broken and
> has been issuing more and more questionable advisories. Various hosted
> projects have been writing security policies to help users know which
> bugs might have security implications.
> 
> https://sourceware.org/cgit/elfutils/tree/SECURITY
> https://sourceware.org/cgit/binutils-gdb/tree/binutils/SECURITY.txt
> https://gcc.gnu.org/cgit/gcc/tree/SECURITY.txt
> 
> The glibc project even setup their own security mailing list and CNA
> (CVE Numbering Authority) publishing their own advisories:
> https://sourceware.org/glibc/security.html
> https://sourceware.org/cgit/glibc/tree/advisories
> 
> To double check that generated files in source repositories are really
> what was intended the container builders now have an autotools
> generated files checker, autoregen, for gcc, binutils and gdb:
> https://inbox.sourceware.org/20231115194803.GW31613@gnu.wildebeest.org/
> 
> Sourceware hosts were not affected by the xz-backdoor. But we did
> reset the https://builder.sourceware.org containers of debian-testing,
> fedora-rawhide and opensuse-tumbleweed. These containers however
> didn't have ssh installed, were running on isolated VMs on separate
> machines from our main hosts, snapshots and backup servers.
> 
> We introduced an "aging inactive users" policy. Accounts are now
> automatically disabled when not used for a year (after a warning).
> https://inbox.sourceware.org/overseers/ZhCho2hjRACDztxy@elastic.org
> 
> = New and upgraded hardware
> 
> There have been complaints about overloaded builders on
> https://builder.sourceware.org. So OSUOSL have provided us with
> another arm64 and x86_64 server. The new servers do the larger gcc and
> glibc builds so the other builders can do quicker (smaller) CI builds
> without having to wait on the big jobs.
> 
> StarFive has donated 4 VisionFive-2 RISC-V boards with 8GB, 4-core
> JH7110 supporting the RV64GC ISA for https://builder.sourceware.org/
> Which has allowed us to setup CI (and try) builders for various
> projects: annobin, binutils(+try), bzip2, debugedit, dwz,
> elfutils(+try), glibc, gdb, poke, and libabigail(+try).
> 
> One of the drives in server2 broke down. It was part of a 10 drive
> raid6 setup, which can take 2 bad disks before full failure. We also
> have a full mirror on server3, which has a similar raid6 setup. We
> ordered 3 new disks, one as replacement for the bad disk and a spare
> for server2 and server3 in case of future drive failures. The drive
> has been replaced and everything is running smoothly again.
> 
> Thanks to Red Hat server2 got a RAM upgrade to 512G.
> 
> = Finances
> 
> To create a hardware replacement fund we setup
> https://sourceware.org/donate.html
> 
> There were $5.500+ in individual donations in the last year.
> 
> And Valgrind was picked for a FUTO https://futo.org Microgrant, which
> has been donated to Sourceware through the Software Freedom
> Conservancy for maintaining and expanding the infrastructure for
> Valgrind and other core toolchain and developer tool projects.
> FUTO then doubled their contribution to $2.000.
> 
> Thanks to our hardware and services partners we didn't have much
> direct expenses. We spend ~$300 on the replacement disks and $20 on
> domain registration.
> 
> Total income was $7,611.73, total expenses were $321.76.
> Note that income is after currency conversions and administration costs.
> 
> Which leaves us with $7,289.97 for our current hardware replacement fund.
> 
> = Next year plans
> 
> To prepare for next year we held various open office and public email
> discussions with the community and made plans for Sourceware and the
> hosted projects secure software development frameworks.
> 
> https://inbox.sourceware.org/20240325100226.GL5673@gnu.wildebeest.org
> https://inbox.sourceware.org/20240401150617.GF19478@gnu.wildebeest.org
> https://inbox.sourceware.org/20240417232725.GC25080@gnu.wildebeest.org
> 
> After the xz-backdoor incident obviously a lot of discussions focused
> on various security aspects. The Sourceware Project Leadership
> Committee turned those ideas into concrete plans for next year:
> 
> Secure Sourceware Project Goals
> https://sourceware.org/sourceware-security-vision.html Secure
> 
> More isolation of existing services. Modernizing account
> processes. Release upload process improvements. Hardware keys for
> administrators, release managers and developers. Pull-request
> server. Part time junior system administrator.
> 
> We are currently working with the Conservancy to fund these plans.
> 
> = Conclusion
> 
> This first year as a Conservancy Member Project has been really good
> for Sourceware and we hope to continue the relationship for many years
> to come. We urge the community to support the Software Freedom
> Conservancy by becoming a Conservancy Sustainer
> https://sfconservancy.org/sustainer

Re: Sourceware @ Conservancy - Year One

[Mark Wielaard]

Hi Maxim,

On Thu, May 30, 2024 at 12:18:38PM +0400, Maxim Kuvyrkov via Overseers wrote:
> > On May 29, 2024, at 23:02, Mark Wielaard <mark@klomp.org> wrote:
> > And a special thanks to ARM who have been using
> > https://patchwork.sourceware.org/ to provide a pre-commit testing
> > service for various projects.
> 
> Thanks for the great update!
> 
> Minor nitpick: pre-commit testing for AArch64 and AArch32
> architectures is provided by Linaro Toolchain Working Group (Linaro
> TCWG).

Sorry for getting the credit wrong. Proper credit is important. And in
this case I really should have known. All pre-commit emails start with
[Linaro-TCWG-CI]. I did think about just mentioning the individuals
who made things happen. But then getting individual names wrong is
even worse than getting corporation names wrong...

Thanks Maxim for making the Linaro Toolchain Working Group pre-commit
testing for AArch64 and AArch32 happen!

Cheers,

Mark