[Michael Matz] Re: Don't let search bots look at buglist.cgi

[Michael Matz] Re: Don't let search bots look at buglist.cgi

[Ian Lance Taylor]

[-- Attachment #1: Type: text/plain, Size: 41 bytes --]

fche: a comment about MnoGoSearch.

Ian


[-- Attachment #2: Type: message/rfc822, Size: 3077 bytes --]

From: Michael Matz <matz@suse.de>
To: Ian Lance Taylor <iant@google.com>
Cc: Richard Guenther <richard.guenther@gmail.com>, Andrew Haley <aph@redhat.com>, gcc-patches@gcc.gnu.org
Subject: Re: Don't let search bots look at buglist.cgi
Date: Tue, 17 May 2011 13:12:23 +0200 (CEST)
Message-ID: <Pine.LNX.4.64.1105171307390.1989@wotan.suse.de>

Hi,

On Mon, 16 May 2011, Ian Lance Taylor wrote:

> >>> httpd being in the top-10 always, fiddling with bugzilla URLs? 
> >>> (Note, I don't have access to gcc.gnu.org, I'm relaying info from 
> >>> multiple instances of discussion on #gcc and richi poking on it; 
> >>> that said, it still might not be web crawlers, that's right, but 
> >>> I'll happily accept
> >>> _any_ load improvement on gcc.gnu.org, how unfounded they might seem)
> 
> I think that simply blocking buglist.cgi has dropped bugzilla off the 
> immediate radar. It also seems to have lowered the load, although I'm 
> not sure if we are still keeping historical data.

Btw. FWIW, I had a quick look at one of the httpd log files, and in seven 
hours on last Saturday (from 5:30 to 12:30), there were overall 435203 GET 
requests, and 391319 of them came from our own MnoGoSearch engine, that's 
90%.  Granted many are then in fact 304 (not modified) responses, but 
still, perhaps the eagerness of our own crawler can be turned down a bit.


Ciao,
Michael.

Re: [Michael Matz] Re: Don't let search bots look at buglist.cgi

[Christopher Faylor]

On Tue, May 17, 2011 at 07:26:01AM -0700, Ian Lance Taylor wrote:
>Btw. FWIW, I had a quick look at one of the httpd log files, and in seven 
>hours on last Saturday (from 5:30 to 12:30), there were overall 435203 GET 
>requests, and 391319 of them came from our own MnoGoSearch engine, that's 
>90%.  Granted many are then in fact 304 (not modified) responses, but 
>still, perhaps the eagerness of our own crawler can be turned down a bit.

Isn't MnoGoSearch doing what it's supposed to be doing?  I don't see why we'd
turn it down.