[postmaster@chiark.greenend.org.uk: [postmaster@sources.redhat.com] Excessive retries by your mail system]

[postmaster@chiark.greenend.org.uk: [postmaster@sources.redhat.com] Excessive retries by your mail system]

[Christopher Faylor]

Anyone have a clue on if this is correct and, if so, how to fix it?

cgf

----- Forwarded message from Ian Jackson as chiark postmaster <postmaster@chiark.greenend.org.uk> -----

From: Ian Jackson as chiark postmaster <postmaster@chiark.greenend.org.uk>
To: postmaster@sources.redhat.com
Subject: [postmaster@sources.redhat.com] Excessive retries by your mail system
Date: Wed, 13 Aug 2003 15:10:22 +0100

In the hour 1220-1320 UTC today (which I'm picking as a convenient
example), sources.redhat.com made 114 connections to my system
chiark.greenend.org.uk, 102 of which were rejected by my system with a
421 banner message (due to your system's excessive use of concurrent
SMTP sessions and its excessive history of SMTP errors when talking to
mine).

That's an average of one failed connection attempt every 35 seconds.
This is grossly excessive.  It's much faster than the retry rates
recommended in RFC1123 (Host Requirements).  It is also a much faster
retry rate than I have configured my system to permit to a single
calling site.

You are triggering capacity reservation and rate-limiting mechanisms
which are intended to cope with denial-of-service attacks and to slow
down spammers.  As a result the real mail which ought to be flowing
from your system to mine (various mailing lists hosted on
sources.redhat.com) is suffering delays.

Please could you reconfigure your system to retry much less often.
See RFC1123 s5.3.1.  Your system appears to be in violation (for
example) of the following paragraph, for example:

      The sender MUST delay retrying a particular destination
      after one attempt has failed.  In general, the retry
      interval SHOULD be at least 30 minutes; however, more
      sophisticated and variable strategies will be beneficial
      when the sender-SMTP can determine the reason for non-
      delivery.

When you've made your system stop hammering mine so much, the mail
should start flowing normally within an hour or two.

In the meantime you will probably not be able to reply by email.  If
you want to get in touch with me, please phone me on +44 1223 723614.
If I don't hear from you by this time tomorrow I'll try looking you up
in whois or the like.

Thanks for your attention.

Regards,
Ian Jackson.

----- End forwarded message -----

Re: [postmaster@chiark.greenend.org.uk: [postmaster@sources.redhat.com] Excessive retries by your mail system]

[Matthew Galgoci]

I'd start by looking at the mail logs to see what it was trying to 
deliver. There may have been multiple messages (like from a high
volume list) which would rule out his theory about the retries being
too quick and against rfc.

On Mon, 18 Aug 2003, Christopher Faylor wrote:

> Anyone have a clue on if this is correct and, if so, how to fix it?
> 
> cgf
> 
> ----- Forwarded message from Ian Jackson as chiark postmaster <postmaster@chiark.greenend.org.uk> -----
> 
> From: Ian Jackson as chiark postmaster <postmaster@chiark.greenend.org.uk>
> To: postmaster@sources.redhat.com
> Subject: [postmaster@sources.redhat.com] Excessive retries by your mail system
> Date: Wed, 13 Aug 2003 15:10:22 +0100
> 
> In the hour 1220-1320 UTC today (which I'm picking as a convenient
> example), sources.redhat.com made 114 connections to my system
> chiark.greenend.org.uk, 102 of which were rejected by my system with a
> 421 banner message (due to your system's excessive use of concurrent
> SMTP sessions and its excessive history of SMTP errors when talking to
> mine).
> 
> That's an average of one failed connection attempt every 35 seconds.
> This is grossly excessive.  It's much faster than the retry rates
> recommended in RFC1123 (Host Requirements).  It is also a much faster
> retry rate than I have configured my system to permit to a single
> calling site.
> 
> You are triggering capacity reservation and rate-limiting mechanisms
> which are intended to cope with denial-of-service attacks and to slow
> down spammers.  As a result the real mail which ought to be flowing
> from your system to mine (various mailing lists hosted on
> sources.redhat.com) is suffering delays.
> 
> Please could you reconfigure your system to retry much less often.
> See RFC1123 s5.3.1.  Your system appears to be in violation (for
> example) of the following paragraph, for example:
> 
>       The sender MUST delay retrying a particular destination
>       after one attempt has failed.  In general, the retry
>       interval SHOULD be at least 30 minutes; however, more
>       sophisticated and variable strategies will be beneficial
>       when the sender-SMTP can determine the reason for non-
>       delivery.
> 
> When you've made your system stop hammering mine so much, the mail
> should start flowing normally within an hour or two.
> 
> In the meantime you will probably not be able to reply by email.  If
> you want to get in touch with me, please phone me on +44 1223 723614.
> If I don't hear from you by this time tomorrow I'll try looking you up
> in whois or the like.
> 
> Thanks for your attention.
> 
> Regards,
> Ian Jackson.
> 
> ----- End forwarded message -----
> 

-- 

Matthew Galgoci		"Dirty deeds, done dirt cheap"
System Administrator
Red Hat, Inc
919.754.3700 x44155

Re: [postmaster@chiark.greenend.org.uk: [postmaster@sources.redhat.com] Excessive retries by your mail system]

[Christopher Faylor]

On Mon, Aug 18, 2003 at 11:47:57AM -0400, Matthew Galgoci wrote:
>I'd start by looking at the mail logs to see what it was trying to
>deliver.  There may have been multiple messages (like from a high
>volume list) which would rule out his theory about the retries being
>too quick and against rfc.

Um, yes.  I assumed that this was a consequence of the way qmail sends
messages.  I was wondering if anyone had specific experience with this,
if there were qmail settings for this, and if his interpretation of
RFC1123 was correct.

In particular, I was hoping that Ian or Jason might have an idea
about how this relates to the qmail setup on sources.redhat.com.

cgf

Re: [postmaster@chiark.greenend.org.uk: [postmaster@sources.redhat.com] Excessive retries by your mail system]

[Ian Lance Taylor]

> From: Ian Jackson as chiark postmaster <postmaster@chiark.greenend.org.uk>
> Subject: [postmaster@sources.redhat.com] Excessive retries by your mail system
> To: postmaster@sources.redhat.com
> Date: Wed, 13 Aug 2003 15:10:22 +0100
> 
> 
> In the hour 1220-1320 UTC today (which I'm picking as a convenient
> example), sources.redhat.com made 114 connections to my system
> chiark.greenend.org.uk, 102 of which were rejected by my system with a
> 421 banner message (due to your system's excessive use of concurrent
> SMTP sessions and its excessive history of SMTP errors when talking to
> mine).
> 
> That's an average of one failed connection attempt every 35 seconds.
> This is grossly excessive.  It's much faster than the retry rates
> recommended in RFC1123 (Host Requirements).  It is also a much faster
> retry rate than I have configured my system to permit to a single
> calling site.
> 
> You are triggering capacity reservation and rate-limiting mechanisms
> which are intended to cope with denial-of-service attacks and to slow
> down spammers.  As a result the real mail which ought to be flowing
> from your system to mine (various mailing lists hosted on
> sources.redhat.com) is suffering delays.
> 
> Please could you reconfigure your system to retry much less often.
> See RFC1123 s5.3.1.  Your system appears to be in violation (for
> example) of the following paragraph, for example:
> 
>       The sender MUST delay retrying a particular destination
>       after one attempt has failed.  In general, the retry
>       interval SHOULD be at least 30 minutes; however, more
>       sophisticated and variable strategies will be beneficial
>       when the sender-SMTP can determine the reason for non-
>       delivery.
> 
> When you've made your system stop hammering mine so much, the mail
> should start flowing normally within an hour or two.

sources.redhat.com runs qmail.  qmail does implement the restriction
above.  However, since sources.redhat.com sends so many messages to so
many failing hosts, and since qmail records a fixed number of hosts
which have failed and updates the list in a circular fashion, it is
possible that the list of hosts is being overwritten such that qmail
is trying your system when it normally would not.

That said, the fact is that sources.redhat.com generates a great deal
of mail, and if people on your system are signed up to several of the
high volume mailing lists, those mailing lists will consistently
trigger rate-limiting mechanisms.  We've seen this before on other
systems.  The effect is that the e-mail traffic from
sources.redhat.com will never flow smoothly in the presence of rate
limitations, and, indeed, sources.redhat.com will start bouncing
e-mail messages sent to your users, and will eventually simply remove
them from the mailing lists.

I'm afraid there is no particularly good way around this.  In general,
high volume mailing lists and rate limitations are not compatible.
You should either make an exception to your rate limiting code for
sources.redhat.com, or you should prohibit your users from signing up
for sources.redhat.com mailing lists (which include gcc.gnu.org
mailing lists).

Ian