From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 10135 invoked by alias); 17 May 2011 16:42:37 -0000 Received: (qmail 10126 invoked by uid 22791); 17 May 2011 16:42:36 -0000 X-SWARE-Spam-Status: No, hits=-1.4 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY X-Spam-Check-By: sourceware.org Received: from nm8-vm0.bullet.mail.sp2.yahoo.com (HELO nm8-vm0.bullet.mail.sp2.yahoo.com) (98.139.91.194) by sourceware.org (qpsmtpd/0.43rc1) with SMTP; Tue, 17 May 2011 16:42:19 +0000 Received: from [98.139.91.65] by nm8.bullet.mail.sp2.yahoo.com with NNFMP; 17 May 2011 16:42:19 -0000 Received: from [98.136.185.43] by tm5.bullet.mail.sp2.yahoo.com with NNFMP; 17 May 2011 16:42:19 -0000 Received: from [127.0.0.1] by smtp104.mail.gq1.yahoo.com with NNFMP; 17 May 2011 16:42:18 -0000 Received: from cgf.cx (cgf@173.48.46.160 with login) by smtp104.mail.gq1.yahoo.com with SMTP; 17 May 2011 09:42:18 -0700 PDT X-Yahoo-SMTP: jenXL62swBAWhMTL3wnej93oaS0ClBQOAKs8jbEbx_o- Received: from localhost (ednor.casa.cgf.cx [192.168.187.5]) by cgf.cx (Postfix) with ESMTP id 4C0C742804C; Tue, 17 May 2011 12:42:17 -0400 (EDT) Date: Tue, 17 May 2011 16:42:00 -0000 From: Christopher Faylor To: Ian Lance Taylor , overseers@gcc.gnu.org Subject: Re: [Michael Matz] Re: Don't let search bots look at buglist.cgi Message-ID: <20110517164216.GA32319@ednor.casa.cgf.cx> Mail-Followup-To: Ian Lance Taylor , overseers@gcc.gnu.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Mailing-List: contact overseers-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: , Sender: overseers-owner@sourceware.org X-SW-Source: 2011-q2/txt/msg00075.txt.bz2 On Tue, May 17, 2011 at 07:26:01AM -0700, Ian Lance Taylor wrote: >Btw. FWIW, I had a quick look at one of the httpd log files, and in seven >hours on last Saturday (from 5:30 to 12:30), there were overall 435203 GET >requests, and 391319 of them came from our own MnoGoSearch engine, that's >90%. Granted many are then in fact 304 (not modified) responses, but >still, perhaps the eagerness of our own crawler can be turned down a bit. Isn't MnoGoSearch doing what it's supposed to be doing? I don't see why we'd turn it down.