Re: bugs not showing up on Google

[frederik@ofb.net]

> > joseph@codesourcery.com suggested that I email you about my
> > observation that most of your bugs are not showing up on Google.
> > [...]
> 
> I don't know about "most"; undoubtedly many appear and some do not.
> It may be relevant that we have had to throttle googlebot from
> full access to the sourceware web servers because it was repeatedly
> found ignoring robots.txt and saturating the server with traffic.
> So we have reluctantly slowed its access down.  I expect it to
> get around to all the bugzilla entries over time, just maybe not as
> fast as you expect.

Thanks Frank for your reply. The entry I was looking at was over a
year old. I don't know what you mean by "over time" but I would
consider that too long. Also I don't think it would take that long for
even a throttled Googlebot to crawl your site.

I'm not sure how a crawler is supposed to see all the bugs, is there a
way of listing them all without going through a search form?

Apparently there are ways to enforce robots.txt using mod_rewrite: as
long as Googlebot doesn't change its user agent, I think you can more
or less easily prevent it from accessing a given URL:

https://perishablepress.com/eight-ways-to-blacklist-with-apaches-mod_rewrite/comment-page-4/

That seems easier to me than QoS tuning.

Even better would be if we could report bugs to Google but ... yeah.
For me it's always been a Wall of Silence.

By the way, I couldn't find a public archive of this mailing list,
should we be discussing this on Bugzilla in case other Bugzilla
maintainers want to benefit from your experience?

https://sourceware.org/bugzilla/show_bug.cgi?id=23581

Maybe I can paste these messages into a comment on that bug and then
add overseers to the Cc list? Or am I tripping and no one cares?

Thanks,

Frederick