Collecting information about duplicated messages

Collecting information about duplicated messages

[Tulio Magno Quites Machado Filho]

Hi,

I'm trying to collect information about a message sent to libc-alpha that
got duplicated (known as IDs 69604 and 69605 on ezmlm).

From what I can see in the headers, there is something wrong happening
in the communication of IBM and Sourceware.org:

    Received: from e38.co.us.ibm.com (HELO e38.co.us.ibm.com) (32.97.110.159) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (CAMELLIA256-SHA encrypted) ESMTPS; Mon, 16 May 2016 22:27:37 +0000
 
    Received: from e38.co.us.ibm.com (HELO e38.co.us.ibm.com) (32.97.110.159) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (CAMELLIA256-SHA encrypted) ESMTPS; Mon, 16 May 2016 22:37:38 +0000

Do you happen to have any logs about this message that could give me any hints
on what happened?

Could you provide me a temporary mailing list in order to try to reproduce
it and catch the error in the wild?

Thank you in advance,

-- 
Tulio Magno

Re: Collecting information about duplicated messages

[Joseph Myers]

On Fri, 20 May 2016, Tulio Magno Quites Machado Filho wrote:

> "Frank Ch. Eigler" <fche@redhat.com> writes:
> 
> >> But, I'd appreciate if you could take a look at the logs and send me
> >> whatever information you have about the transmission of that message [1].
> >
> > From the logs here, it simply looks like two separate messages sent a
> > few seconds apart, as if someone may have fat-fingered "git
> > send-email".  The SMTP parts of the logs appear to have ben aged away
> > already; increasing their retention limits now.
> 
> OK.  Unfortunately, it wasn't a case of fat fingers.  :-(

And it's also happened before.  E.g. 
<https://sourceware.org/ml/libc-alpha/2015-06/msg00788.html> was a large 
message that appeared several times on the list, probably for the same 
reasons.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Collecting information about duplicated messages

[Tulio Magno Quites Machado Filho]

"Frank Ch. Eigler" <fche@redhat.com> writes:

>> But, I'd appreciate if you could take a look at the logs and send me
>> whatever information you have about the transmission of that message [1].
>
> From the logs here, it simply looks like two separate messages sent a
> few seconds apart, as if someone may have fat-fingered "git
> send-email".  The SMTP parts of the logs appear to have ben aged away
> already; increasing their retention limits now.

OK.  Unfortunately, it wasn't a case of fat fingers.  :-(

From what I already know, the IBM server didn't receive an answer for the
<CRLF>.<CRLF> and timed out at 22:29:09 UTC.

Assuming these servers' time are synchronized, this is order of the events:

22:27:32 e38.co.us.ibm.com stamps the header of the message.  Everything
         before this is identical.

22:27:37 sourceware.org stamps the header of the message.

22:29:07 sourceware.org stamps the header of the message after spam check.

22:29:09 e38.co.us.ibm.com times out and put the message on the queue again.

22:37:38 sourceware.org stamps the header of the 2nd message.

22:38:28 sourceware.org stamps the header of the 2nd message after spam
         check.

22:38:29 e38.co.us.ibm.com logs the 2nd message was successfully delivered.

So, it's possible that something prevented IBM from receiving the answer
to <CRLF>.<CRLF>.  But I'm afraid the answer might be related to the fact
that a 500kB message took around 100 seconds to be delivered.
Maybe we had a network issue?  Or was one of the servers under high load?

-- 
Tulio Magno

Re: Collecting information about duplicated messages

[Frank Ch. Eigler]

Hi -

> But, I'd appreciate if you could take a look at the logs and send me
> whatever information you have about the transmission of that message [1].

From the logs here, it simply looks like two separate messages sent a
few seconds apart, as if someone may have fat-fingered "git
send-email".  The SMTP parts of the logs appear to have ben aged away
already; increasing their retention limits now.

- FChE

Re: Collecting information about duplicated messages

[Tulio Magno Quites Machado Filho]

Tulio Magno Quites Machado Filho <tuliom@linux.vnet.ibm.com> writes:

> Do you happen to have any logs about this message that could give me any hints
> on what happened?
>
> Could you provide me a temporary mailing list in order to try to reproduce
> it and catch the error in the wild?

After thinking more on this: the temporary mailing list isn't necessary yet.
I'll use my own alias for these tests first.  Hopefully, I'll be able
to reproduce it this way.

But, I'd appreciate if you could take a look at the logs and send me
whatever information you have about the transmission of that message [1].

Thanks!

[1] https://sourceware.org/ml/libc-alpha/2016-05/msg00320.html

-- 
Tulio Magno