Something's hammering cvsweb.

Something's hammering cvsweb.

[Phil Edwards]

After someone complained of CVS refusals due to the load being too high,
I looked briefly and found that scads of cvsweb.cgi processes had driven
the load into the high 50's.  Here's a snippet:

apache   25646  6.0  0.1  5356 3592 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20010122
apache   25647  8.6  0.1  5244 3480 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20001030
apache   25649  8.6  0.1  5356 3592 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20010115
apache   25684  6.6  0.1  4976 3188 ?        R    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20000417
apache   25685  6.3  0.1  4980 3192 ?        R    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=hammer-3_3-merge-20030414
apache   25692  8.3  0.1  5276 3512 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20001106
apache   25693  8.0  0.1  5236 3472 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=merged-arm-thumb-backend-merge_20000113
apache   25694  9.0  0.1  5352 3588 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20010625
apache   25696 10.0  0.1  4976 3188 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20000717
apache   25698 10.0  0.1  4976 3188 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=egcs_ss_980214


Note the start time; they're all being fetched simultaneously.  (My guess
is some bizarre spambot trying to do every combination of form-fillout
with the drop-down list of tags.)  At any rate, it's a very effective DoS.


-- 
LUKE:  Is Perl better than Python?
YODA:  No... no... no.  Quicker, easier, more seductive.
LUKE:  But how will I know why Python is better than Perl?
YODA:  You will know.  When your code you try to read six months from now.

Re: Something's hammering cvsweb.

[Jonathan Larmour]

Phil Edwards wrote:
> After someone complained of CVS refusals due to the load being too high,
> I looked briefly and found that scads of cvsweb.cgi processes had driven
> the load into the high 50's.  Here's a snippet:
[snip]
> 
> Note the start time; they're all being fetched simultaneously.  (My guess
> is some bizarre spambot trying to do every combination of form-fillout
> with the drop-down list of tags.)  At any rate, it's a very effective DoS.

I'm distracted enough to have a peek out of curiousity: here's an entry 
from gcc.gnu.org's log:
61.250.89.149 - - [29/Oct/2003:07:36:59 +0000] "GET 
/cgi-bin/cvsweb.cgi/gcc/install-sh?only_with_tag=ra-merge-20020521 
HTTP/1.0" 200 143991 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; 
DigExt; empas)"

The address is assigned to "jobkorea.co.kr" whoever they are, and DigExt 
can be found mentioned here which indicates it's a web crawler 
http://www.webmasterworld.com/forum11/2141.htm and more specifically: 
"digext is the crawler for IE's "make available offline" mode. Also known 
as subscriptions."

Another thing says: "DigExt is a common string that the IEAK
(Internet Explorer Administration Kit) put in when you customise an install,"

Certainly other people indicate it hammers servers. Some people wrote 
throttling scripts to detect this type of thing. But I guess someone will 
be along shortly to bar them entirely instead ;).

Jifl
-- 
eCosCentric    http://www.eCosCentric.com/    The eCos and RedBoot experts
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine

Re: Something's hammering cvsweb.

[Christopher Faylor]

On Wed, Oct 29, 2003 at 12:29:31AM -0500, Phil Edwards wrote:
>After someone complained of CVS refusals due to the load being too high,
>I looked briefly and found that scads of cvsweb.cgi processes had driven
>the load into the high 50's.  Here's a snippet:
>
>apache   25646  6.0  0.1  5356 3592 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20010122
>apache   25647  8.6  0.1  5244 3480 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20001030
>apache   25649  8.6  0.1  5356 3592 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20010115
>apache   25684  6.6  0.1  4976 3188 ?        R    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20000417
>apache   25685  6.3  0.1  4980 3192 ?        R    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=hammer-3_3-merge-20030414
>apache   25692  8.3  0.1  5276 3512 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20001106
>apache   25693  8.0  0.1  5236 3472 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=merged-arm-thumb-backend-merge_20000113
>apache   25694  9.0  0.1  5352 3588 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20010625
>apache   25696 10.0  0.1  4976 3188 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=gcc_ss_20000717
>apache   25698 10.0  0.1  4976 3188 ?        S    05:23   0:00 /usr/bin/perl -s /var/www/gcc/cgi-bin/cvsweb.cgi only_with_tag=egcs_ss_980214
>
>
>Note the start time; they're all being fetched simultaneously.  (My guess
>is some bizarre spambot trying to do every combination of form-fillout
>with the drop-down list of tags.)  At any rate, it's a very effective DoS.

It probably is a spambot since the robots.txt file is supposed to disallow
access to /cgi-bin/.  I don't see any indication of a problem now but I'll
look over the logs later today.

In the meantime, I've tightened up the robots.txt on gcc.gnu.org to disallow
some of the more popular spambot user agents.

cgf