bugs not showing up on Google

bugs not showing up on Google

[frederik]

Dear Overseers,

joseph@codesourcery.com suggested that I email you about my
observation that most of your bugs are not showing up on Google. I
opened a bug against Bugzilla but he suggests that is invalid and that
I need to email instead.

https://sourceware.org/bugzilla/show_bug.cgi?id=23581

I had thought that a bugzilla bug would be a good way for people to
talk about this, but whatever - it's up to you! I'm just a dude from
the Internet.

Best wishes,

Frederick

Re: bugs not showing up on Google

[Frank Ch. Eigler]

Hi -


> joseph@codesourcery.com suggested that I email you about my
> observation that most of your bugs are not showing up on Google.
> [...]

I don't know about "most"; undoubtedly many appear and some do not.
It may be relevant that we have had to throttle googlebot from
full access to the sourceware web servers because it was repeatedly
found ignoring robots.txt and saturating the server with traffic.
So we have reluctantly slowed its access down.  I expect it to
get around to all the bugzilla entries over time, just maybe not as
fast as you expect.

- FChE

Re: bugs not showing up on Google

[frederik]

> > joseph@codesourcery.com suggested that I email you about my
> > observation that most of your bugs are not showing up on Google.
> > [...]
> 
> I don't know about "most"; undoubtedly many appear and some do not.
> It may be relevant that we have had to throttle googlebot from
> full access to the sourceware web servers because it was repeatedly
> found ignoring robots.txt and saturating the server with traffic.
> So we have reluctantly slowed its access down.  I expect it to
> get around to all the bugzilla entries over time, just maybe not as
> fast as you expect.

Thanks Frank for your reply. The entry I was looking at was over a
year old. I don't know what you mean by "over time" but I would
consider that too long. Also I don't think it would take that long for
even a throttled Googlebot to crawl your site.

I'm not sure how a crawler is supposed to see all the bugs, is there a
way of listing them all without going through a search form?

Apparently there are ways to enforce robots.txt using mod_rewrite: as
long as Googlebot doesn't change its user agent, I think you can more
or less easily prevent it from accessing a given URL:

https://perishablepress.com/eight-ways-to-blacklist-with-apaches-mod_rewrite/comment-page-4/

That seems easier to me than QoS tuning.

Even better would be if we could report bugs to Google but ... yeah.
For me it's always been a Wall of Silence.

By the way, I couldn't find a public archive of this mailing list,
should we be discussing this on Bugzilla in case other Bugzilla
maintainers want to benefit from your experience?

https://sourceware.org/bugzilla/show_bug.cgi?id=23581

Maybe I can paste these messages into a comment on that bug and then
add overseers to the Cc list? Or am I tripping and no one cares?

Thanks,

Frederick

Re: bugs not showing up on Google

[Christopher Faylor]

On Wed, Sep 05, 2018 at 11:01:17PM -0700, frederik@ofb.net wrote:
>By the way, I couldn't find a public archive of this mailing list,

That's because this is a restricted mailing list.  There is a public
archive but it's not advertised.

cgf