mysql/patchwork overload

mysql/patchwork overload

[Mark Wielaard]

Hi,

I haven't done a full analyzis yet, but from aprox 03:53 to 04:33 UTC
mysqld was completely overloaded taking 20+ cores are 100%. At the
same time patchwork was sending 1000+ emails to the patchwork admins
about being unable to access mysql. So I assume this was a spider/bot
attack on patchwork. But it also took out bugzilla.

For now all I did was stop patchwork, restart mysql and then started
patchwork again. This seems to have resolved the issue for now.

I also added one specific ip address 47.76.35.19 (Alibaba Cloud) to
the block.include list that did about ~20.000 requests to patchwork
around that time.

The same ip address was blocked earlier (December 8th) because it was
doing a great amount of bugzilla buglist queries. In the Christmas
spirit it was unblocked a few days ago. But I cannot say I am feeling
the Christmas spirit right now.

It might just have been lots of requests, but an analysis of which
specific queries cause mysqld to use up so much cpu time would be
helpful.

Merry Christmas,

Mark

Re: mysql/patchwork overload

[Mark Wielaard]

On Tue, Dec 26, 2023 at 06:28:04AM +0100, Mark Wielaard via Overseers wrote:
> I haven't done a full analyzis yet, but from aprox 03:53 to 04:33 UTC
> mysqld was completely overloaded taking 20+ cores are 100%. At the
> same time patchwork was sending 1000+ emails to the patchwork admins
> about being unable to access mysql. So I assume this was a spider/bot
> attack on patchwork. But it also took out bugzilla.

We don't seem to keep a log of slow mysqld queries, and I don't have
root user access to the database. But I think I found the issue(s)
looking at the httpd logs.

First there was one ipv6 (block) 2404:c140:1f00:8::0/64 that abused
the bugzilla rest api. The queries timed out (after a minute) with a
504, but still seemed to keep mysql busy taking up 1 core for tens of
minutes. I put it on the block.include list.

Second bots seemed to keep hitting the patchwork lists using different
orderings, searching for different submitters, delegates, etc. I added
the following to the robots.txt:

commit d11ea11bfa1cbbeb84423d707d58445a41b0ff21
Author: Mark Wielaard <mark@klomp.org>
Date:   Wed Dec 27 19:35:38 2023 +0000

    robots.txt add user, register, mail and various list params

diff --git a/htdocs/robots.txt b/htdocs/robots.txt
index 72c192e..b47f80a 100644
--- a/htdocs/robots.txt
+++ b/htdocs/robots.txt
@@ -1,3 +1,11 @@
 User-Agent: *
 Disallow: /api/
+Disallow: /user/
+Disallow: /register/
+Disallow: /mail/
+Disallow: /project/*/list/?*order=
+Disallow: /project/*/list/?*series=
+Disallow: /project/*/list/?*submitter=
+Disallow: /project/*/list/?*delegate=
+Disallow: /project/*/list/?*param=
 Crawl-Delay: 5