Re: mysql/patchwork overload

[Mark Wielaard <mark@klomp.org>]

On Tue, Dec 26, 2023 at 06:28:04AM +0100, Mark Wielaard via Overseers wrote:
> I haven't done a full analyzis yet, but from aprox 03:53 to 04:33 UTC
> mysqld was completely overloaded taking 20+ cores are 100%. At the
> same time patchwork was sending 1000+ emails to the patchwork admins
> about being unable to access mysql. So I assume this was a spider/bot
> attack on patchwork. But it also took out bugzilla.

We don't seem to keep a log of slow mysqld queries, and I don't have
root user access to the database. But I think I found the issue(s)
looking at the httpd logs.

First there was one ipv6 (block) 2404:c140:1f00:8::0/64 that abused
the bugzilla rest api. The queries timed out (after a minute) with a
504, but still seemed to keep mysql busy taking up 1 core for tens of
minutes. I put it on the block.include list.

Second bots seemed to keep hitting the patchwork lists using different
orderings, searching for different submitters, delegates, etc. I added
the following to the robots.txt:

commit d11ea11bfa1cbbeb84423d707d58445a41b0ff21
Author: Mark Wielaard <mark@klomp.org>
Date:   Wed Dec 27 19:35:38 2023 +0000

    robots.txt add user, register, mail and various list params

diff --git a/htdocs/robots.txt b/htdocs/robots.txt
index 72c192e..b47f80a 100644
--- a/htdocs/robots.txt
+++ b/htdocs/robots.txt
@@ -1,3 +1,11 @@
 User-Agent: *
 Disallow: /api/
+Disallow: /user/
+Disallow: /register/
+Disallow: /mail/
+Disallow: /project/*/list/?*order=
+Disallow: /project/*/list/?*series=
+Disallow: /project/*/list/?*submitter=
+Disallow: /project/*/list/?*delegate=
+Disallow: /project/*/list/?*param=
 Crawl-Delay: 5