From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id 2CBB33858D37; Wed, 27 Dec 2023 19:44:32 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2CBB33858D37 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 2CBB33858D37 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=45.83.234.184 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1703706274; cv=none; b=N7bJCIqrn7IdqdscQR+EYqbH/m2Fetol4OpnH1lWujs0I79Io5lF79RngqOYLUuy8u6DQmr/j/HfBpP2Ku5XYG1WLa9uH5PE/okmI+cI+B3ZG5//E0EITAHHBeKYZMhlHEC6WaVQLOGTfv22mAh1qzcX1k4gE8CVn9wgXfMwRiE= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1703706274; c=relaxed/simple; bh=9tlnxi+fZ0I5Z7KwCY/Kkhql3eQOpn4aV65LGwyFtzQ=; h=Date:From:To:Subject:Message-ID:MIME-Version; b=woC6W1mhx8WSuu4k7Yx/hHnKsxPwlPXqxBJ0IHtTigx3FwTnajylOA43Rmo5Ajo/VtjiGTIKdqrVWwL1numLZuEkdQ92upbvzGrSlyUNoYJWd+2SfStPtX6h6P5bsJC2Fe3QpqmUWHK5btcGKMY0vZneK7r6WbRdhzH9A2aAxMI= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by gnu.wildebeest.org (Postfix, from userid 1000) id 1D4EC302BB42; Wed, 27 Dec 2023 20:44:31 +0100 (CET) Date: Wed, 27 Dec 2023 20:44:31 +0100 From: Mark Wielaard To: Mark Wielaard via Overseers Cc: patchwork-admin@sourceware.org Subject: Re: mysql/patchwork overload Message-ID: <20231227194431.GM6896@gnu.wildebeest.org> References: <20231226052804.GI6896@gnu.wildebeest.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231226052804.GI6896@gnu.wildebeest.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-8.7 required=5.0 tests=BAYES_00,GIT_PATCH_0,JMQ_SPF_NEUTRAL,KAM_DMARC_STATUS,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Tue, Dec 26, 2023 at 06:28:04AM +0100, Mark Wielaard via Overseers wrote: > I haven't done a full analyzis yet, but from aprox 03:53 to 04:33 UTC > mysqld was completely overloaded taking 20+ cores are 100%. At the > same time patchwork was sending 1000+ emails to the patchwork admins > about being unable to access mysql. So I assume this was a spider/bot > attack on patchwork. But it also took out bugzilla. We don't seem to keep a log of slow mysqld queries, and I don't have root user access to the database. But I think I found the issue(s) looking at the httpd logs. First there was one ipv6 (block) 2404:c140:1f00:8::0/64 that abused the bugzilla rest api. The queries timed out (after a minute) with a 504, but still seemed to keep mysql busy taking up 1 core for tens of minutes. I put it on the block.include list. Second bots seemed to keep hitting the patchwork lists using different orderings, searching for different submitters, delegates, etc. I added the following to the robots.txt: commit d11ea11bfa1cbbeb84423d707d58445a41b0ff21 Author: Mark Wielaard Date: Wed Dec 27 19:35:38 2023 +0000 robots.txt add user, register, mail and various list params diff --git a/htdocs/robots.txt b/htdocs/robots.txt index 72c192e..b47f80a 100644 --- a/htdocs/robots.txt +++ b/htdocs/robots.txt @@ -1,3 +1,11 @@ User-Agent: * Disallow: /api/ +Disallow: /user/ +Disallow: /register/ +Disallow: /mail/ +Disallow: /project/*/list/?*order= +Disallow: /project/*/list/?*series= +Disallow: /project/*/list/?*submitter= +Disallow: /project/*/list/?*delegate= +Disallow: /project/*/list/?*param= Crawl-Delay: 5