Sourceware @ Conservancy - Year Two

Sourceware @ Conservancy - Year Two

[Mark Wielaard]

Sourceware joined Conservancy as a member project on May 15 2023
https://sfconservancy.org/news/2023/may/15/sourceware-joins-sfc/

Sourceware has provided the infrastructure for core toolchain and
developer tool projects for more than 25 years.
https://sourceware.org/sourceware-25-roadmap.html

Conservancy has helped us turn from a purely volunteer into a
professional organization with an eight person strong Project
Leadership Committee, monthly open office hours, multiple hardware
services partners, expanded services, and a more diverse funding model
that allows us to enter into contracts with paid contractors or staff
when appropriate.

It was again a busy year, so we would like to summarize what happened
last year and our plans for the next one.

- Communications and the Big User Survey
- A Sourceware Forge (an experiment with Forgejo)
- AI/LLM scraperbots attacks and Anubis
- Cyber Security and Regulations
- New and upgraded hardware
- Finances, thanks for the donations
- Next year plans, more, bigger servers
- Conclusion and thank you

= Communications and the Big User Survey

  In the last year we organized 12 Open Office meetings on IRC and on
  the Software Freedom Conservancy's Big Blue Button instance
  https://bbb.sfconservancy.org/

  The SFC also extendeds the use of their BBB server to any Sourceware
  project that wants to host video meetings.
  https://sourceware.org/mission.html#organization

  Sourceware infrastructure community quarterly updates were posted
  for:

  24Q2
    https://inbox.sourceware.org/20240605164429.GG12557@gnu.wildebeest.org
  24Q3
    https://inbox.sourceware.org/20240930222921.GL3393@gnu.wildebeest.org
  24Q4
    https://inbox.sourceware.org/20241220130604.GJ25993@gnu.wildebeest.org
  25Q1
    https://inbox.sourceware.org/20250422230422.GI2323@gnu.wildebeest.org

  The Sourceware Big User Survey 2025 ran from Friday, 14 March to
  Monday, 31 March. We got 103 responses with a nice mix of
  developers, users and maintainers from various hosted projects.

  Full results can be found at https://sourceware.org/survey-2025

  Thanks to everybody who responded, this really helps the Sourceware
  Project Leadership Committee decide how to allocate resources.

  Sourceware PLC members and Conservancy Staff were also present at the
  Cauldron 2024 and Fosdem 2025 conferences.

  Sourceware @ Cauldron 2024 BoF Report, Topics, and Notes:
  https://inbox.sourceware.org/20240925004343.GR21963@gnu.wildebeest.org/

  Sourceware also regularly posts updates on infrastructure issue on
  the fediverse at @sourceware@fosstodon.org
  https://fosstodon.org/@sourceware

= A Sourceware Forge (an experiment with Forgejo)

  In multiple discussions at the Cauldron various developers and
  maintainers indicated they really would like to do a serious
  experiment with a Forge and a pull-request workflow.
  https://forge.sourceware.org

  We secured a VM from Red Hat OSCI that should have enough resources
  for the initial experiment. The Sourceware PLC will discuss what
  resources are needed if we want to roll this out for all Sourceware
  projects. We already made an estimate for a larger gitolite server
  as part of the Security Vision document:
  https://sourceware.org/sourceware-security-vision.html
  Part of the Forgejo experiment will be making sure the resource
  estimates are correct.

  Sergio and Mark created the initial setup, which is almost fully
  scripted, but still has to be done by hand:
  https://sourceware.org/cgit/forge/tree/SETUP
  Claudio has been turning this into a fully automated Ansible setup.
  https://inbox.sourceware.org/20250207200803.10136-1-claudio.bantaloukas@arm.com/
  And Richard setup a GCC wiki page to track all issues:
  https://gcc.gnu.org/wiki/ForgeExperiment

= AI/LLM scraperbots attacks and Anubis

  Sourceware has been fighting the new AI/LLM scraperbots since start
  of this calendar year. We are not alone in this.

  https://lwn.net/Articles/1008897/
  https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/

  The first couple of months we have tried to isolate services more
  and block various ip-blocks that were abusing the servers. But that
  only helped so much. Unfortunately the scraper bots are using lots
  of ip addresses (probably by installing "free" VPN services that use
  normal user connections as exit point) and pretending to be common
  browsers/agents.

  So we ended up "protecting" most services with Anubis
  https://anubis.techaro.lol/

  This helped enormously to block almost all scraperbots. The downside
  is that normal users must solve a quick javascript challenge or
  change their browser agent to not get challenged.
  
  Having isolated most services we managed to not require anubis for
  any static content. But when using patchwork, bunsen, bugzilla,
  gitweb, cgit or the wikis you might now have to enable javascript or
  change your browser user agent. This should not impact any scripts,
  just browsers (or bots pretending to be browsers).  If it does cause
  trouble, please let us know.

= Cyber Security and Regulations

  Aging inactive users policy. Every 3 months we now run the "aging
  inactive users" process by sending emails to users without any
  activity in the last year. And then disabled accounts that really
  weren't active (putting them in the emeritus group).
  https://inbox.sourceware.org/ZhQZXogZMozVjIYn@elastic.org

  Please keep your account details up to date so that we always have a
  way of contacting you. Please see the account management page on how
  to set your current email address:
  https://sourceware.org/sourceware/accountinfo.html

  Sourceware Cyber Security FAQ. After lots of discussions at some of
  our Open Office hours, at the Cauldron, with other Software Freedom
  organizations and some of our hardware and services providers we
  created an Sourceware Cyber Security FAQ "explainer" about topics
  like the "US Improving the Nation's Cybersecurity Executive Order
  14028", "EU Cyber Resilience Act (EU CRA)" and "Secure Software
  Development Framework (NIST SP 800-218)" or who is required to use
  Zero Trust (NIST SP 800-207) cloud-computing environments.

  https://sourceware.org/cyber-security-faq.html

  We also added a section with Recommendations for Sourceware hosted
  projects. For Sourceware hosted projects that want to have a
  documented verifiable cybersecurity policy we now have a policy
  checklist your project can follow. Most are common sense things most
  projects already do.
  https://sourceware.org/cyber-security-faq.html#policy-checklist

  Somewhat related the Software Freedom Conservancy published a blog
  post about the recent bans of Russian contributors in the Linux
  project and whether Free Software projects need to worry about
  U.S. Sanctions.

  https://sfconservancy.org/blog/2024/dec/12/linux-banned-russian-contributors-do-i-need-to/

  Signed-commit census report. Each quarter we now publish how each
  project is doing with signed git commits and the percentage of users
  that sign their commits.

= New and upgraded hardware

  Thanks to RISC-V International and SOPHGO we got a Milk-V Pioneer
  Box for builder.sourceware.org that has been used for gcc CI.

  When originally setup a full gcc build and check took ~10 hours.
  After various bug fixes and tweaks to the build system it now takes
  ~4 hours. It has 64 cores, but single core performance isn't very
  fast. So fixing parallelism bottlenecks saved a lot of build time.
  https://inbox.sourceware.org/20240801210720.GQ24765@gnu.wildebeest.org/

  Also thanks to RISC-V International we got 3 more buildbot CI workers.
  One HiFive Premier P550
  https://www.sifive.com/boards/hifive-premier-p550
  and two Banana Pi BPI-F3
  https://wiki.banana-pi.org/Banana_Pi_BPI-F3

  They have been used for testing the Valgrind risc-v backend that
  will was introduced with Valgrind 3.25.0.

  The P550 now runs a gdb and full testsuite build. One bpi-f3 runs
  glibc and the full testsuite. The other bpi-f3 runs a gcc bootstrap
  and full testsuite the bpi-f3 has an 8 core SpacemiT K1 supporting
  rvv 1.0.

  Unfortunately we had to shut down the Pioneer box, which was faster
  than the above machines, but just overheated too often and then
  needed manual intervention. It was used up to the GCC 15.1 release
  to make sure there were no build time regressions.

= Finances, thanks for the donations

  Our total income from personal donations was ~$3000+, around ~$250 a
  month, but not evenly distributed over the months.

  We had some trouble with paypal that caused not everybody who
  donated this calendar year having received a thank you note. Our
  apologies for that. We hope the paypal import system will be fixed
  soon, so everybody receives their thank you email.

  Thanks to our hardware and services partners we didn't have much
  direct expenses.

  And there were no domain fees this year since those were payed up
  for more than a year last time. We will get some this and next
  calendar year though.

  We did pay paypal bankcharges and other fees of around ~$60.

  And because last year we did had to replace some disks we did buy a
  couple of extra spare disks for ~$160.

  In summary, we started with $7289.97 from last year, added ~$3000+
  from donations, and payed paypal bankcharges and other fees of ~$60
  and ~$160 for spare disks. Leaving us with $10095.15 at the end of
  our second year.

= Next year plans, more, bigger servers

  Somewhere in Q3 2025 the Red Hat community cage, which hosts two of
  our servers, will move to another data center
  https://www.osci.io/tenants/

  The PLC wants to take advantage of this move by using some of our
  current budget to add a bigger machine in the new datacenter. The
  new machine will be installed and configured before the move of the
  other two servers. Making the switch as smooth as possible. And it
  will help with our goal to isolate more services on separate
  machines or VMs.

  We are also still looking for sponsors to accelerate some of our
  other (security) plans:
  https://sourceware.org/sourceware-security-vision.html#plans

= Conclusion and thank you

  The first two years as a Conservancy Member Project has been really
  good for Sourceware and we hope to continue the relationship for
  many years to come. We urge the community to support the Software
  Freedom Conservancy by becoming a Conservancy Sustainer
  https://sfconservancy.org/sustainer

  The OSUOSL is an important partner for Sourceware which hosts
  various servers for us. Helping OSUOSL helps not only Sourceware but
  lots of other Free Software projects:
  https://osuosl.org/blog/osl-future-update/
  https://osuosl.org/donate/

  Please see https://sourceware.org/donate.html if you want to
  financially support Sourceware directly.

  Don't forget that there are lots of projects that Sourceware and all
  hosted projects rely on. If possible sent them a thank you.
  
  Like Bugzilla https://www.bugzilla.org/donate
  And Forgejo https://liberapay.com/forgejo

  Some are also Software Freedom Conservancy members, like buildbot,
  git and xapian https://sfconservancy.org/projects/current/
  
The Sourceware PLC,

 Frank Ch. Eigler, Christopher Faylor, Ian Kelling, Ian Lance Taylor,
 Tom Tromey, Jon Turney, Mark J. Wielaard, Elena Zannoni