Re: [Bug web/?????] New: Fwd: failure notice: Bugzilla down.

Re: [Bug web/?????] New: Fwd: failure notice: Bugzilla down.

[Martin Sebor]

On 08/14/2017 04:22 PM, Eric Gallager wrote:
> I'm emailing this manually to the list because Bugzilla is down and I
> can't file a bug on Bugzilla about Bugzilla being down. The error
> message looks like this:

Bugzilla and the rest of gcc.gnu.org have been down much of
the afternoon/evening due to a hard drive upgrade (the old disk
apparently failed).  You're not the only one who found out about
it the hard way.  I (and I suspect most others) also discovered
it when things like Git and SVN (and Bugzilla) stopped working.

I've CC'd the gcc list to let others know (not sure what list
to subscribe to in order to get a heads up on these kinds of
maintenance issues).

Martin

>
> Software error:
>
> Can't connect to the database.
> Error: Can't connect to local MySQL server through socket
> '/var/lib/mysql/mysql.sock' (2)
>   Is your database installed and up and running?
>   Do you have the correct username and password selected in localconfig?
>
>
> For help, please send mail to the webmaster
> (sourcemaster@sourceware.org), giving this error message and the time
> and date of the error.
>
> Unfortunately, the email address listed for the webmaster,
> <sourcemaster@sourceware.org>, sends me its own error message when I
> try to email it, because the address apparently doesn't exist. I'm
> forwarding a copy of the failure email with this message. Is there a
> different address I should be emailing instead?
>
> Thanks,
> Eric Gallager
>
> ---------- Forwarded message ----------
> From: MAILER-DAEMON@sourceware.org
> Date: 14 Aug 2017 22:03:54 -0000
> Subject: failure notice
> To: egall@gwmail.gwu.edu
>
> Hi. This is the qmail-send program at sourceware.org.
> I'm afraid I wasn't able to deliver your message to the following addresses.
> This is a permanent error; I've given up. Sorry it didn't work out.
>
> <sourcemaster@sourceware.org>:
> Sorry, no mailbox here by that name. (#5.1.1)
>
> --- Below this line is a copy of the message.
>
> Return-Path: <egall@gwmail.gwu.edu>
> Received: (qmail 24677 invoked by uid 89); 14 Aug 2017 22:03:52 -0000
> Authentication-Results: sourceware.org; auth=none
> X-Virus-Checked: by ClamAV 0.99.2 on sourceware.org
> X-Virus-Found: No
> X-Spam-SWARE-Status: No, score=0.5 required=5.0
> tests=RCVD_IN_DNSWL_NONE,RCVD_IN_SORBS_SPAM,SPF_PASS autolearn=no
> version=3.3.2 spammy=
> X-Spam-Status: No, score=0.5 required=5.0
> tests=RCVD_IN_DNSWL_NONE,RCVD_IN_SORBS_SPAM,SPF_PASS autolearn=no
> version=3.3.2
> X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on sourceware.org
> X-Spam-Level:
> X-HELO: mail-wm0-f42.google.com
> Received: from mail-wm0-f42.google.com (HELO mail-wm0-f42.google.com)
> (74.125.82.42)
>  by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon,
> 14 Aug 2017 22:03:51 +0000
> Received: by mail-wm0-f42.google.com with SMTP id i66so3361599wmg.0
>         for <sourcemaster@sourceware.org>; Mon, 14 Aug 2017 15:03:50 -0700 (PDT)
> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
>         d=gwmail-gwu-edu.20150623.gappssmtp.com; s=20150623;
>         h=mime-version:from:date:message-id:subject:to;
>         bh=A6iFIzlnSazSObmM6Wb7tDW+ZAb4ojY1zG6bxpdCLCQ=;
>         b=cvRTbJxAB1UKTO01xqOGTtvgLjbLLOumX/FUdM3ruLkvMpJx6QiO7X51cU6X6jcx24
>          ReoVRxaW8NwWcVJts9JB5tPJSg/8+ljL8ywMoa9Aa6S1MQ4PS246aVgKM5aUNmaB1E+u
>          NrQMAmsxixJG/wmrAnSpQ/imIt7XNHRBNnZ5/B1JiGAt3ChjHwTNKV/9W4fxydsJL9BE
>          iL9D/duAl6C/RAvBPQ5BC0Kv+/HtzhH8bwgElbjP93hMJk/nAfceI7Rh/4zIgY5V6ZHS
>          E7LzrUetB7aeyzdlTy6i/3Lsyb0wnUuocZdZtGndQD24WSngk3fInlHwWrcUVdTpRMQ5
>          e4tg==
> X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
>         d=1e100.net; s=20161025;
>         h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
>         bh=A6iFIzlnSazSObmM6Wb7tDW+ZAb4ojY1zG6bxpdCLCQ=;
>         b=GQbdg3EVSV+aMfKeSffGxae54jos3nNUt3qQWoSthUOEFy57YAeV5Fn86IOYLTw4fI
>          gZKi7samoFHs0yHdF4sawi+xVMfZMzGPqGrJjsAIHYSE9T+o9vx0cEJPhPs0rswqinCO
>          WovmT8Kh1jNEaIyxLwZGk07lt5oyWTk72zaT+CxWjamKbLJzDfqWMfpahDuTVI74QQie
>          m8qtMTBQjek1Di51mxiBcuaLlr1xNqwrVS3JQEjdl2NCOBEDodLosGkUJCFR29hw6EAK
>          ag3Lo96yUeNlfjOXv8/FYbCTrbcrhxbms02n0p3oK3cSHkoqZKcUOQQJzuNq0c38W9PU
>          6A0w==
> X-Gm-Message-State: AHYfb5ilSTxPFsi8yTAJsSJ6LpZu+DahzJygxoF6Q5usa2PIRZDQhD5i
> 	b5akwEZnwfnttac6yAR/os3Kbj3Gqwbh
> X-Received: by 10.80.215.68 with SMTP id i4mr25610854edj.258.1502748228995;
>  Mon, 14 Aug 2017 15:03:48 -0700 (PDT)
> MIME-Version: 1.0
> Received: by 10.80.129.227 with HTTP; Mon, 14 Aug 2017 15:03:48 -0700 (PDT)
> From: Eric Gallager <egall@gwmail.gwu.edu>
> Date: Mon, 14 Aug 2017 18:03:48 -0400
> Message-ID: <CAMfHzOvVKr2AJ3d=j8iF64bWAsRgKO-u+2CwqNRwmuv=YY1qrQ@mail.gmail.com>
> Subject: GCC bugzilla down?
> To: sourcemaster@sourceware.org
> Content-Type: text/plain; charset="UTF-8"
>
> When I tried to access the GCC Bugzilla this evening of August 14,
> 2017, at 6PM Eastern Daylight Time, it displayed this error message:
>
> Software error:
>
> Can't connect to the database.
> Error: Can't connect to local MySQL server through socket
> '/var/lib/mysql/mysql.sock' (2)
>   Is your database installed and up and running?
>   Do you have the correct username and password selected in localconfig?
>
>
> For help, please send mail to the webmaster
> (sourcemaster@sourceware.org), giving this error message and the time
> and date of the error.
>
> So that's this email. Is the database okay?
>
> Thanks,
> Eric Gallager
>

Re: [Bug web/?????] New: Fwd: failure notice: Bugzilla down.

[Martin Sebor]

On 08/15/2017 07:27 AM, Jonathan Wakely wrote:
> On 15 August 2017 at 04:10, Martin Sebor wrote:
>> On 08/14/2017 04:22 PM, Eric Gallager wrote:
>>>
>>> I'm emailing this manually to the list because Bugzilla is down and I
>>> can't file a bug on Bugzilla about Bugzilla being down. The error
>>> message looks like this:
>
>
> Even if it were possible, there wouldn't be any point in filing a bug
> that bugzilla's down, and so not much point emailing gcc-bugs either
> (since that's for bugzilla mail).  Using gcc@ seems like the right
> list to me.
>
> N.B. since the server is being restored from a backup all of
> yesterday's changes to bugzilla have been lost, including all Richi's
> 7.2 release changes, and Eric's housekeeping.
>
> I don't suggest redoing all that work until all services are fully
> restored, in case it's lost again.

It looks like the data loss extends beyond 8/14.  Bug 81840
was created Sunday afternoon but is not in the database:

   https://gcc.gnu.org/ml/gcc-bugs/2017-08/msg01303.html

(Strangely, 81841 is there, as is 81839.)

Is there a plan to restore the lost data somehow or should
we try to do that ourselves for the bugs we know about?

Martin

Re: [Bug web/?????] New: Fwd: failure notice: Bugzilla down.

[Martin Sebor]

On 08/15/2017 10:27 AM, Joseph Myers wrote:
> On Tue, 15 Aug 2017, Martin Sebor wrote:
>
>> It looks like the data loss extends beyond 8/14.  Bug 81840
>> was created Sunday afternoon but is not in the database:
>>
>>   https://gcc.gnu.org/ml/gcc-bugs/2017-08/msg01303.html
>>
>> (Strangely, 81841 is there, as is 81839.)
>
> That's another 81839 replacing the original 81839.  As I noted on
> overseers, the cut-off for GCC Bugzilla appears to be between
> <https://gcc.gnu.org/ml/gcc-bugs/2017-08/msg01289.html> and
> <https://gcc.gnu.org/ml/gcc-bugs/2017-08/msg01290.html>, early Sunday
> morning UTC (and for sourceware Bugzilla it may well be similar).

Thanks for the update.  Can I subscribe to this list?

(It might be helpful to post an announcement of what's going on
here for those of us who aren't subscribed to the overseers list
or even are aware of its archives.)

Martin

Re: [Bug web/?????] New: Fwd: failure notice: Bugzilla down.

[Joseph Myers]

On Tue, 15 Aug 2017, Martin Sebor wrote:

> On 08/15/2017 10:27 AM, Joseph Myers wrote:
> > On Tue, 15 Aug 2017, Martin Sebor wrote:
> > 
> > > It looks like the data loss extends beyond 8/14.  Bug 81840
> > > was created Sunday afternoon but is not in the database:
> > > 
> > >   https://gcc.gnu.org/ml/gcc-bugs/2017-08/msg01303.html
> > > 
> > > (Strangely, 81841 is there, as is 81839.)
> > 
> > That's another 81839 replacing the original 81839.  As I noted on
> > overseers, the cut-off for GCC Bugzilla appears to be between
> > <https://gcc.gnu.org/ml/gcc-bugs/2017-08/msg01289.html> and
> > <https://gcc.gnu.org/ml/gcc-bugs/2017-08/msg01290.html>, early Sunday
> > morning UTC (and for sourceware Bugzilla it may well be similar).
> 
> Thanks for the update.  Can I subscribe to this list?

I believe it's open subscription.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [Bug web/?????] New: Fwd: failure notice: Bugzilla down.

[Joseph Myers]

On Wed, 16 Aug 2017, NightStrike wrote:

> On Mon, Aug 14, 2017 at 11:10 PM, Martin Sebor <msebor@gmail.com> wrote:
> > On 08/14/2017 04:22 PM, Eric Gallager wrote:
> >>
> >> I'm emailing this manually to the list because Bugzilla is down and I
> >> can't file a bug on Bugzilla about Bugzilla being down. The error
> >> message looks like this:
> >
> > Bugzilla and the rest of gcc.gnu.org have been down much of
> > the afternoon/evening due to a hard drive upgrade (the old disk
> > apparently failed).  You're not the only one who found out about
> 
> There's no RAID?

The problem was apparently a kernel bug (in the 2.6.32 RHEL6 kernel, may 
or may not be in current kernels) that arose when adding a new SSD mirror 
to the RAID with the old hard drives marked writemostly.  (Separately, one 
of the hard drives in the existing HDD RAID is showing signs of failing 
and needs to be replaced.)

-- 
Joseph S. Myers
joseph@codesourcery.com