I noticed that buglist.cgi was taking quite a bit of CPU time.  I looked
at some of the long running instances, and they were coming from
searchbots.  I can't think of a good reason for this, so I have
committed this patch to the gcc.gnu.org robots.txt file to not let
searchbots search through lists of bugs.  I plan to make a similar
change on the sourceware.org and cygwin.com sides.  Please let me know
if this seems like a mistake.

Does anybody have any experience with
http://code.google.com/p/bugzilla-sitemap/ ?  That might be a slightly
better approach.

Ian