[Bug web/53632] New: [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[Bug web/53632] New: [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

             Bug #: 53632
           Summary: [bugzilla] Bugzilla being very slow to submit changes,
                    sending duplicate emails
    Classification: Unclassified
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: web
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: redi@gcc.gnu.org
                CC: LpSolit@netscape.net


For the past week or two Bugzilla has been very slow to submit updates or new
bugs, and when several changes happen to a PR in quick succession the emails
contain multiple updates and so duplicate information.

Here there are three emails each containing the same three changes:
http://gcc.gnu.org/ml/gcc-bugs/2012-06/msg00561.html
http://gcc.gnu.org/ml/gcc-bugs/2012-06/msg00563.html
http://gcc.gnu.org/ml/gcc-bugs/2012-06/msg00564.html

Here there are three emails for two changes:
http://gcc.gnu.org/ml/gcc-bugs/2012-06/msg00005.html
http://gcc.gnu.org/ml/gcc-bugs/2012-06/msg00006.html
http://gcc.gnu.org/ml/gcc-bugs/2012-06/msg00007.html

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[LpSolit at netscape dot net]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #1 from Frédéric Buclin <LpSolit at netscape dot net> 2012-06-11 11:04:57 UTC ---
I did no changes to the Bugzilla code for almost two months, so I guess the
problem is external to Bugzilla. Based on duplicated comments in emails, I
suspect the mail server is slow these days, and so subsequent changes to bugs
happen before all emails had the time to be sent. In that case, Bugzilla
assumes all emails should be resent to avoid some email loss.

Is there some problem with the SMTP server these days (e.g. heavy load)? This
would explain it.

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[LpSolit at netscape dot net]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #2 from Frédéric Buclin <LpSolit at netscape dot net> 2012-06-11 11:06:57 UTC ---
Wow, Bugzilla is indeed very slow, even when sending two emails only.

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #3 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-06-11 11:16:38 UTC ---
Yeah :-)  It's got _really_ slow. I don't remember which day it happened, but
it was a very noticeable sudden change from reasonably responsive to very slow.

After updating bugzilla in one browser tab/window if you hit refresh on the
gcc-bugs list archive (http://gcc.gnu.org/ml/gcc-bugs/current/) in another
tab/window you can see the mail from bugzilla arrive fairly quickly, but it
takes much longer for the bugzilla page to finish "submitting" (in fact I
believe it has finished submitting but is still waiting for something before
reloading the next page.)

I don't have access to the GCC server to see where the delay or load is, but it
could be similar to the wiki-slowness described at
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53257 which is related to delays
sending emails.

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[LpSolit at netscape dot net]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

Frédéric Buclin <LpSolit at netscape dot net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2012-06-11
     Ever Confirmed|0                           |1

--- Comment #4 from Frédéric Buclin <LpSolit at netscape dot net> 2012-06-11 11:25:17 UTC ---
(In reply to comment #3)
> After updating bugzilla in one browser tab/window if you hit refresh on the
> gcc-bugs list archive (http://gcc.gnu.org/ml/gcc-bugs/current/) in another
> tab/window you can see the mail from bugzilla arrive fairly quickly, but it
> takes much longer for the bugzilla page to finish "submitting"

Ah, that's very interesting. I could try to debug this problem, but this means
I would have to spam this bug a bit to see where the slowness comes from. Is
that acceptable?

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[LpSolit at netscape dot net]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #5 from Frédéric Buclin <LpSolit at netscape dot net> 2012-06-11 11:36:20 UTC ---
fche told me on IRC that jobqueue has been disabled two weeks ago, which is
exactly when the slowness started to appear. So it may still be mail-related.
But a SMTP server shouldn't need more than a few seconds to send only 2 emails.
fche also told me that the SMTP server is doing some spam checks for outgoing
emails. I wonder if that could be the reason. Still investigating...

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[LpSolit at netscape dot net]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #6 from Frédéric Buclin <LpSolit at netscape dot net> 2012-06-11 11:46:26 UTC ---
He just told me that the checks for outgoing emails take between 2 and 10
seconds *per email*! I think we found the culprit. Either the scan must be
optimized (e.g. whitelist GCC Bugzilla to not scan emails sent by it) or
jobqueue must be re-enabled. In both cases, I cannot do anything myself.

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[LpSolit at netscape dot net]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #7 from Frédéric Buclin <LpSolit at netscape dot net> 2012-06-11 12:03:19 UTC ---
fche says that some spammers create a Bugzilla account to send spam, which is
why these spam checks are enabled for outgoing emails. So to mitigate the
problem till jobqueue is re-enabled, one alternative could be to whitelist
emails sent by @gcc.gnu.org users, as those users are trusted people. We could
even extend this whitelist to all users having editbugs privs, as you need to
be a trusted user to get such privileges. This way, only user accounts with no
privileges would see the slowness due to these checks, including spammers.

From a technical point of view, the email server should look at the
X-Bugzilla-Who header, which contains the email address of the user responsible
for the email to be sent. If it's a trusted user, just send the email without
scanning it. This header cannot be abused by spammers.

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #8 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-06-11 13:03:46 UTC ---
(In reply to comment #7)
> fche says that some spammers create a Bugzilla account to send spam,

Yes, we've had a few attacks recently.

Thanks for identifying the problem so quickly!

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #9 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-07-03 15:06:56 UTC ---
I don't know if anything was changed, but the speed seems OK now

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[LpSolit at netscape dot net]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #10 from Frédéric Buclin <LpSolit at netscape dot net> 2012-08-14 19:01:12 UTC ---
(In reply to comment #9)
> I don't know if anything was changed, but the speed seems OK now

It seems very slow to me. It took a while to send 3 bugmails only.

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[redi at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #11 from Jonathan Wakely <redi at gcc dot gnu.org> 2012-08-15 14:28:02 UTC ---
yes it was only ok very briefly, and has been slow again since then

[Bug web/53632] [bugzilla] Bugzilla being very slow to submit changes, sending duplicate emails

[pinskia at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53632

--- Comment #13 from Andrew Pinski <pinskia at gcc dot gnu.org> 2013-04-18 16:03:39 UTC ---
I think this was fixed when we migrated to a new machine.  The old machine was
overloaded.