From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <overseers-return-11462-listarch-overseers=sources.redhat.com@sourceware.org>
Received: (qmail 31356 invoked by alias); 16 May 2011 09:45:59 -0000
Received: (qmail 31339 invoked by uid 22791); 16 May 2011 09:45:58 -0000
X-SWARE-Spam-Status: No, hits=-2.3 required=5.0
	tests=AWL,BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST
X-Spam-Check-By: sourceware.org
Received: from mail-ww0-f51.google.com (HELO mail-ww0-f51.google.com) (74.125.82.51)
    by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Mon, 16 May 2011 09:45:44 +0000
Received: by wwf26 with SMTP id 26so4432783wwf.8
        for <multiple recipients>; Mon, 16 May 2011 02:45:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.54.6 with SMTP id o6mr4040613wbg.61.1305539142604; Mon, 16
 May 2011 02:45:42 -0700 (PDT)
Received: by 10.227.38.129 with HTTP; Mon, 16 May 2011 02:45:42 -0700 (PDT)
In-Reply-To: <mcry62a7cau.fsf@coign.corp.google.com>
References: <mcry62a7cau.fsf@coign.corp.google.com>
Date: Mon, 16 May 2011 09:45:00 -0000
Message-ID: <BANLkTim22-m_jqhHCiNiGKVs5km96t6M3w@mail.gmail.com>
Subject: Re: Don't let search bots look at buglist.cgi
From: Richard Guenther <richard.guenther@gmail.com>
To: Ian Lance Taylor <iant@google.com>
Cc: overseers@gcc.gnu.org, gcc-patches@gcc.gnu.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Mailing-List: contact overseers-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <overseers.sourceware.org>
List-Archive: <http://sourceware.org/ml/overseers/>
List-Post: <mailto:overseers@sourceware.org>
List-Help: <mailto:overseers-help@sourceware.org>,
	<http://sourceware.org/ml/#faqs>
Sender: overseers-owner@sourceware.org
X-SW-Source: 2011-q2/txt/msg00068.txt.bz2

On Fri, May 13, 2011 at 7:14 PM, Ian Lance Taylor <iant@google.com> wrote:
> I noticed that buglist.cgi was taking quite a bit of CPU time. =A0I looked
> at some of the long running instances, and they were coming from
> searchbots. =A0I can't think of a good reason for this, so I have
> committed this patch to the gcc.gnu.org robots.txt file to not let
> searchbots search through lists of bugs. =A0I plan to make a similar
> change on the sourceware.org and cygwin.com sides. =A0Please let me know
> if this seems like a mistake.
>
> Does anybody have any experience with
> http://code.google.com/p/bugzilla-sitemap/ ? =A0That might be a slightly
> better approach.

Shouldn't we keep searchbots way from bugzilla completely?  Searchbots
can crawl the gcc-bugs mailinglist archives.

Richard.

> Ian
>
>