Bugzilla timing out

Bugzilla timing out

[Martin Sebor]

I'm having trouble bringing up bugs or updating them.  Has
anyone else noticed Bugzilla (and/or other services running
on gcc.gnu.org) being very slow or timing out?

Thanks
Martin

Re: Bugzilla timing out

[Joseph Myers]

On Mon, 15 Jan 2018, Martin Sebor wrote:

> I'm having trouble bringing up bugs or updating them.  Has
> anyone else noticed Bugzilla (and/or other services running
> on gcc.gnu.org) being very slow or timing out?

I'm presuming this comes from the read errors on sdi that I just reported 
to overseers (if reads keep having to fail / time out before the RAID 
gives up on sdi and reads the data from another disk in the RAID instead).

[710507.350074] sd 0:2:8:0: [sdi] CDB: Read(10): 28 00 07 e0 1c 50 00 00 08 00
[710507.350301] end_request: I/O error, dev sdi, sector 132127824
[710507.529984] md/raid:md3: read error corrected (8 sectors at 132125776 on sdi1)
[710507.530346] md/raid:md3: read error corrected (8 sectors at 132125784 on sdi1)
[710507.530704] md/raid:md3: read error corrected (8 sectors at 132125792 on sdi1)

etc.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Bugzilla timing out

[H.J. Lu]

On Mon, Jan 15, 2018 at 12:58 PM, Joseph Myers <joseph@codesourcery.com> wrote:
> On Mon, 15 Jan 2018, Martin Sebor wrote:
>
>> I'm having trouble bringing up bugs or updating them.  Has
>> anyone else noticed Bugzilla (and/or other services running
>> on gcc.gnu.org) being very slow or timing out?
>
> I'm presuming this comes from the read errors on sdi that I just reported
> to overseers (if reads keep having to fail / time out before the RAID
> gives up on sdi and reads the data from another disk in the RAID instead).
>
> [710507.350074] sd 0:2:8:0: [sdi] CDB: Read(10): 28 00 07 e0 1c 50 00 00 08 00
> [710507.350301] end_request: I/O error, dev sdi, sector 132127824
> [710507.529984] md/raid:md3: read error corrected (8 sectors at 132125776 on sdi1)
> [710507.530346] md/raid:md3: read error corrected (8 sectors at 132125784 on sdi1)
> [710507.530704] md/raid:md3: read error corrected (8 sectors at 132125792 on sdi1)
>

It took a long time to do "svn ci".


-- 
H.J.

Re: Bugzilla timing out

[Frank Ch. Eigler]

msebor wrote:

> I'm having trouble bringing up bugs or updating them.  Has anyone else
> noticed Bugzilla (and/or other services running on gcc.gnu.org) being
> very slow or timing out?

One failed disk in the server has been replaced yesterday.  Please let
us know if problems persist or reoccur.

- FChE

Re: Bugzilla timing out

[Eric Gallager]

On 1/18/18, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> msebor wrote:
>
>> I'm having trouble bringing up bugs or updating them.  Has anyone else
>> noticed Bugzilla (and/or other services running on gcc.gnu.org) being
>> very slow or timing out?
>
> One failed disk in the server has been replaced yesterday.  Please let
> us know if problems persist or reoccur.
>
> - FChE
>

Problems are still occurring for me; Bugzilla gives me 504 Gateway
Time-outs when I try to access it tonight...

Re: Bugzilla timing out

[Frank Ch. Eigler]

Hi -

> Problems are still occurring for me; Bugzilla gives me 504 Gateway
> Time-outs when I try to access it tonight...

OK, we reworked some of the database routine maintenance workload,
e.g., a nightly cleanup pass that was quite likely excessive, and
will keep monitoring.


- FChE

Re: Bugzilla timing out

[Richard Biener]

On Mon, Jan 22, 2018 at 7:08 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Hi -
>
>> Problems are still occurring for me; Bugzilla gives me 504 Gateway
>> Time-outs when I try to access it tonight...
>
> OK, we reworked some of the database routine maintenance workload,
> e.g., a nightly cleanup pass that was quite likely excessive, and
> will keep monitoring.

With all such administrative workloads keep in mind that "night" might
be "day" in another timezone ;)

Richard.

>
> - FChE

Re: Bugzilla timing out

[Frank Ch. Eigler]

Hi -

On Tue, Jan 23, 2018 at 11:26:42AM +0100, Richard Biener wrote:
> [...]
> > OK, we reworked some of the database routine maintenance workload,
> > e.g., a nightly cleanup pass that was quite likely excessive, and
> > will keep monitoring.
> 
> With all such administrative workloads keep in mind that "night" might
> be "day" in another timezone ;)

Sure, but there are only one or two of you ne'erdowells on the wrong
side of the planet. :-)

- FChE

Re: Bugzilla timing out

[Martin Sebor]

On 01/22/2018 11:08 AM, Frank Ch. Eigler wrote:
> Hi -
>
>> Problems are still occurring for me; Bugzilla gives me 504 Gateway
>> Time-outs when I try to access it tonight...
>
> OK, we reworked some of the database routine maintenance workload,
> e.g., a nightly cleanup pass that was quite likely excessive, and
> will keep monitoring.

Thanks for looking into it.  I'm still (or again) seeing very
poor responsiveness.  Right now, bringing up an existing bug
takes an entire minute.

Martin

Re: Bugzilla timing out

[H.J. Lu]

On Fri, Jan 26, 2018 at 1:52 PM, Martin Sebor <msebor@gmail.com> wrote:
> On 01/22/2018 11:08 AM, Frank Ch. Eigler wrote:
>>
>> Hi -
>>
>>> Problems are still occurring for me; Bugzilla gives me 504 Gateway
>>> Time-outs when I try to access it tonight...
>>
>>
>> OK, we reworked some of the database routine maintenance workload,
>> e.g., a nightly cleanup pass that was quite likely excessive, and
>> will keep monitoring.
>
>
> Thanks for looking into it.  I'm still (or again) seeing very
> poor responsiveness.  Right now, bringing up an existing bug
> takes an entire minute.
>

Same here.  When I opened a new bug today, it timed out.  I had to look up
the database to see if I opened it or not.


-- 
H.J.

Re: Bugzilla timing out

[Frank Ch. Eigler]

Hi -

> Thanks for looking into it.  I'm still (or again) seeing very
> poor responsiveness.  Right now, bringing up an existing bug
> takes an entire minute.

There has been quite a burst in activity on sourceware.org over the
last few hours.  Will look further into why, but quite a bit of it may
be anti-spam processing.

https://sourceware.org/grafana/index.html#/dashboard/file/default.json

- FChE

Re: Bugzilla timing out

[Joseph Myers]

On Fri, 26 Jan 2018, Frank Ch. Eigler wrote:

> Hi -
> 
> > Thanks for looking into it.  I'm still (or again) seeing very
> > poor responsiveness.  Right now, bringing up an existing bug
> > takes an entire minute.
> 
> There has been quite a burst in activity on sourceware.org over the
> last few hours.  Will look further into why, but quite a bit of it may
> be anti-spam processing.

Many copies of a 5 MB message have apparently been timing out in 
sourceware's spam processing, resulting in sourceware repeatedly accepting 
the message while the sender's mail server also times out (sooner) and 
keeps resending it.

https://sourceware.org/ml/libc-alpha/2018-01/msg00901.html

(the list archives uniquify based on message-id, but many separate copies 
have been delivered to libc-alpha so far each with its own ezmlm sequence 
number).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Bugzilla timing out

[Frank Ch. Eigler]

Hi -

> Many copies of a 5 MB message have apparently been timing out in 
> sourceware's spam processing, resulting in sourceware repeatedly accepting 
> the message while the sender's mail server also times out (sooner) and 
> keeps resending it. [...]

Thanks for the pointer.  I tightened up some of the timeouts / parallelism
associated with spam filtering.

- FChE

Re: Bugzilla timing out

[Martin Sebor]

On 01/26/2018 04:08 PM, Frank Ch. Eigler wrote:
> Hi -
>
>> Many copies of a 5 MB message have apparently been timing out in
>> sourceware's spam processing, resulting in sourceware repeatedly accepting
>> the message while the sender's mail server also times out (sooner) and
>> keeps resending it. [...]
>
> Thanks for the pointer.  I tightened up some of the timeouts / parallelism
> associated with spam filtering.

Thanks.  It's a lot snappier now.

Martin