Not usable email content encoding

Not usable email content encoding

[Martin Liška]

Hello.

I noticed some emails reaching gcc-patches@gcc.gnu.org use the following quoting:

```
      case UNGT_EXPR:
      case UNGE_EXPR:
      case UNEQ_EXPR:
+    case MEM_REF:
        /* Binary operations evaluating both arguments (increment and
  =09 decrement are binary internally in GCC).  */
        orig_op0 =3D op0 =3D TREE_OPERAND (expr, 0);
@@ -435,6 +436,14 @@ c_fully_fold_internal (tree expr, bool i
  =09      || TREE_CODE (TREE_TYPE (orig_op0)) =3D=3D FIXED_POINT_TYPE)
  =09  && TREE_CODE (TREE_TYPE (orig_op1)) =3D=3D INTEGER_TYPE)
  =09warn_for_div_by_zero (loc, op1);
+      if (code =3D=3D MEM_REF
+=09  && ret !=3D expr
+=09  && TREE_CODE (ret) =3D=3D MEM_REF)
+=09{
+=09  TREE_READONLY (ret) =3D TREE_READONLY (expr);
+=09  TREE_SIDE_EFFECTS (ret) =3D TREE_SIDE_EFFECTS (expr);
+=09  TREE_THIS_VOLATILE (ret) =3D TREE_THIS_VOLATILE (expr);
+=09}
```

It's probably related to the following email tag:
Content-Transfer-Encoding: quoted-printable

The format is problematic when copying a patch.
Email example:
https://gcc.gnu.org/pipermail/gcc-patches/2020-March/542053.html

Thank you,
Martin

Re: Not usable email content encoding

[Richard Earnshaw]

On 16/03/2020 13:45, Martin Liška wrote:
> Hello.
> 
> I noticed some emails reaching gcc-patches@gcc.gnu.org use the following 
> quoting:
> 
> ```
>       case UNGT_EXPR:
>       case UNGE_EXPR:
>       case UNEQ_EXPR:
> +    case MEM_REF:
>         /* Binary operations evaluating both arguments (increment and
>   =09 decrement are binary internally in GCC).  */
>         orig_op0 =3D op0 =3D TREE_OPERAND (expr, 0);
> @@ -435,6 +436,14 @@ c_fully_fold_internal (tree expr, bool i
>   =09      || TREE_CODE (TREE_TYPE (orig_op0)) =3D=3D FIXED_POINT_TYPE)
>   =09  && TREE_CODE (TREE_TYPE (orig_op1)) =3D=3D INTEGER_TYPE)
>   =09warn_for_div_by_zero (loc, op1);
> +      if (code =3D=3D MEM_REF
> +=09  && ret !=3D expr
> +=09  && TREE_CODE (ret) =3D=3D MEM_REF)
> +=09{
> +=09  TREE_READONLY (ret) =3D TREE_READONLY (expr);
> +=09  TREE_SIDE_EFFECTS (ret) =3D TREE_SIDE_EFFECTS (expr);
> +=09  TREE_THIS_VOLATILE (ret) =3D TREE_THIS_VOLATILE (expr);
> +=09}
> ```
> 
> It's probably related to the following email tag:
> Content-Transfer-Encoding: quoted-printable
> 
> The format is problematic when copying a patch.
> Email example:
> https://gcc.gnu.org/pipermail/gcc-patches/2020-March/542053.html
> 
> Thank you,
> Martin

This isn't new.  It's usually the sender that controls that, though it's 
often automatic in their email client.

R.

Re: Not usable email content encoding

[Martin Liška]

On 3/16/20 2:47 PM, Richard Earnshaw wrote:
> This isn't new.  It's usually the sender that controls that, though it's often automatic in their email client.

Jakub told me that he sees:

Ten mail byl:
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

while the email reaching the mailing list has:

Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Martin

Re: Not usable email content encoding

[Florian Weimer]

* Martin Liška:

> It's probably related to the following email tag:
> Content-Transfer-Encoding: quoted-printable
>
> The format is problematic when copying a patch.
> Email example:
> https://gcc.gnu.org/pipermail/gcc-patches/2020-March/542053.html

This is a known issue in the redhat.com email infrastructure.

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -

> I noticed some emails reaching gcc-patches@gcc.gnu.org use the following quoting:
> It's probably related to the following email tag:
> Content-Transfer-Encoding: quoted-printable

This is not something that the mailing list system does.  This
content-transfer-encoding comes from the original email (as evident in
the /var/lib/mailman/archives/private/gcc-patches.mbox/gcc-patches.mbox
file).

- FChE

Re: Not usable email content encoding

[Alexander Monakov]

On Mon, 16 Mar 2020, Martin Liška wrote:

> It's probably related to the following email tag:
> Content-Transfer-Encoding: quoted-printable
> 
> The format is problematic when copying a patch.
> Email example:
> https://gcc.gnu.org/pipermail/gcc-patches/2020-March/542053.html

I'm surprised it's an issue for you: normally your email client
would transform quoted-printable and copying would do the right thing
(i.e. select actual patch contents, without whitespace munging).

Are you trying to copy from the raw message representation?

Alexander

Re: Not usable email content encoding

[Martin Liška]

On 3/16/20 2:54 PM, Alexander Monakov wrote:
> Are you trying to copy from the raw message representation?

Exactly.

Martin

Re: Not usable email content encoding

[Jonathan Wakely]

On Mon, 16 Mar 2020 at 14:13, Martin Liška wrote:
>
> On 3/16/20 2:54 PM, Alexander Monakov wrote:
> > Are you trying to copy from the raw message representation?
>
> Exactly.

That's never been reliable.

Re: Not usable email content encoding

[Segher Boessenkool]

On Mon, Mar 16, 2020 at 04:54:51PM +0300, Alexander Monakov via Gcc wrote:
> On Mon, 16 Mar 2020, Martin Liška wrote:
> 
> > It's probably related to the following email tag:
> > Content-Transfer-Encoding: quoted-printable
> > 
> > The format is problematic when copying a patch.
> > Email example:
> > https://gcc.gnu.org/pipermail/gcc-patches/2020-March/542053.html
> 
> I'm surprised it's an issue for you: normally your email client
> would transform quoted-printable and copying would do the right thing
> (i.e. select actual patch contents, without whitespace munging).
> 
> Are you trying to copy from the raw message representation?

Everyone trying to work with a patch (instead of just the email) always
is working with the raw message.  Just  patch < mbox  or  git-am mbox
for example.

https://gcc.gnu.org/contribute.html says
  It is strongly discouraged to post patches as MIME parts of type
  application/whatever, disposition attachment or encoded as base64 or
  quoted-printable.

(which many people still do not follow, making reviewing their patches
much harder than needed).


Segher

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -

> > Are you trying to copy from the raw message representation?
> 
> Everyone trying to work with a patch (instead of just the email) always
> is working with the raw message.  Just  patch < mbox  or  git-am mbox
> for example.
> 
> https://gcc.gnu.org/contribute.html says
>   It is strongly discouraged to post patches as MIME parts of type
>   application/whatever, disposition attachment or encoded as base64 or
>   quoted-printable.
> 
> (which many people still do not follow, making reviewing their patches
> much harder than needed).

The key here is to realize that the raw message is not what you get
back from the mailing list reflector, and also not the raw message
that was sent by the sender.  In this day of mta intermediaries,
proxies, reflectors, it may be time to revisit that suggestion.

Attachments seem to survive unmolested better than mail email bodies,
for example.  And MUAs can undo some of the unexpected transformations
in the main body.

- FChE

Re: Not usable email content encoding

[Segher Boessenkool]

Hi!

On Tue, Mar 17, 2020 at 03:51:58PM -0400, Frank Ch. Eigler via Gcc wrote:
> > > Are you trying to copy from the raw message representation?
> > 
> > Everyone trying to work with a patch (instead of just the email) always
> > is working with the raw message.  Just  patch < mbox  or  git-am mbox
> > for example.
> > 
> > https://gcc.gnu.org/contribute.html says
> >   It is strongly discouraged to post patches as MIME parts of type
> >   application/whatever, disposition attachment or encoded as base64 or
> >   quoted-printable.
> > 
> > (which many people still do not follow, making reviewing their patches
> > much harder than needed).
> 
> The key here is to realize that the raw message is not what you get
> back from the mailing list reflector,

Really?  It does not change any of this, afaik?  It never was like that
before, and if it now is, that makes this whole mailman experiment a
failure, imnsho.

> and also not the raw message
> that was sent by the sender.  In this day of mta intermediaries,
> proxies, reflectors, it may be time to revisit that suggestion.

Not really, no.  Any encoding is bad for most workflows.  MIME types
application/* or */x-* do not belong on mailing lists, by definition.
And attachments are (or should be!) just a last resort option to get
your patches through unmolested (and even then it does not always
work for everyone).

Of course most people use attachments with a bad MIME type or
disposition, making it harder to work with still.

> Attachments seem to survive unmolested better than mail email bodies,
> for example.  And MUAs can undo some of the unexpected transformations
> in the main body.

Yes, but not most.

People doing Free Software just need to learn how to use the various
tools involved correctly and effectively.  Email is one such tool.
There is no better one yet (all of the other code review options and
discussion platforms have much bigger downsides than upsides).  We
have been using this since forever, too, and we know how it works
(*including* this encoding/disposition/etc. problem, which has been
around for decades).


Segher

Re: Not usable email content encoding

[Florian Weimer]

* Frank Ch. Eigler via Gcc:

>> > Are you trying to copy from the raw message representation?
>> 
>> Everyone trying to work with a patch (instead of just the email) always
>> is working with the raw message.  Just  patch < mbox  or  git-am mbox
>> for example.
>> 
>> https://gcc.gnu.org/contribute.html says
>>   It is strongly discouraged to post patches as MIME parts of type
>>   application/whatever, disposition attachment or encoded as base64 or
>>   quoted-printable.
>> 
>> (which many people still do not follow, making reviewing their patches
>> much harder than needed).
>
> The key here is to realize that the raw message is not what you get
> back from the mailing list reflector, and also not the raw message
> that was sent by the sender.  In this day of mta intermediaries,
> proxies, reflectors, it may be time to revisit that suggestion.

But these largely are new problems.  It used to work flawlessly.
Patch reencoding problems go back to the redhat.com changes last
November (I understand the responsible vendor is working on a fix, but
I'm not up-to-date on the current developments).

Since the sourceware.org Mailman migration, the From: header is being
rewritten, without any compelling reason.  I certainly do not do any
DMARC checking here, so the rewriting does not benefit me.

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -

> > The key here is to realize that the raw message is not what you get
> > back from the mailing list reflector, and also not the raw message
> > that was sent by the sender.  In this day of mta intermediaries,
> > proxies, reflectors, it may be time to revisit that suggestion.
> 
> But these largely are new problems.  It used to work flawlessly.

I understand that's frustrating.  But these workflows were counting on
literally unspecified behaviours not changing, or outright standards
violations continuing.

> Patch reencoding problems go back to the redhat.com changes last
> November (I understand the responsible vendor is working on a fix,
> but I'm not up-to-date on the current developments).

This one is a standards-compliant reencoding.  Even if mimecast (?)
stops doing it, we can't be sure nothing else will.

> Since the sourceware.org Mailman migration, the From: header is being
> rewritten, without any compelling reason.  I certainly do not do any
> DMARC checking here, so the rewriting does not benefit me.

It benefits you because more and more email services are rejecting or
interfering with mail that is not clean enough.  If you want to
receive mail reliably, or send and have confidence that it is
received, clean mail benefits you.


- FChE

Re: Not usable email content encoding

[Florian Weimer]

* Frank Ch. Eigler:

> Hi -
>
>> > The key here is to realize that the raw message is not what you get
>> > back from the mailing list reflector, and also not the raw message
>> > that was sent by the sender.  In this day of mta intermediaries,
>> > proxies, reflectors, it may be time to revisit that suggestion.
>> 
>> But these largely are new problems.  It used to work flawlessly.
>
> I understand that's frustrating.  But these workflows were counting on
> literally unspecified behaviours not changing, or outright standards
> violations continuing.

Delivery of each individual message is unspecified as well, but we
still count it to happen.  I'm sorry, but I think this argument sounds
a bit vacuous.  With our own infrastructure, we should be able to get
it to behave in the way we need.

>> Patch reencoding problems go back to the redhat.com changes last
>> November (I understand the responsible vendor is working on a fix,
>> but I'm not up-to-date on the current developments).
>
> This one is a standards-compliant reencoding.  Even if mimecast (?)
> stops doing it, we can't be sure nothing else will.

But it's rather unusual in the RFC 822 world, especially with the
decline of sendmail and its peculiar 8BITMIME handling.  I understand
that you get a lot of this in the corporate mail world originally
influenced by X.400, but such messages are rarely handled by
sourceware these days, probably because people rather use Gmail than
wrestle with their inadequate corporate email.

>> Since the sourceware.org Mailman migration, the From: header is being
>> rewritten, without any compelling reason.  I certainly do not do any
>> DMARC checking here, so the rewriting does not benefit me.
>
> It benefits you because more and more email services are rejecting or
> interfering with mail that is not clean enough.  If you want to
> receive mail reliably, or send and have confidence that it is
> received, clean mail benefits you.

It's definitely not cleaner after Mailman has applied its destructive
header rewriting.  It just replaces one address spoofing with another.

What are the plans for Mailman on sourceware?  Will it be replaced
soon with something else, given that the software stack it runs on is
effectively EOL?

It should not be too hard to add a configuration option where
subscribers can opt out of header rewriting, but with Mailman's
upstream status, I'm not sure if that's a worthwhile effort.

Re: Not usable email content encoding

[Michael Matz]

Hi,

On Wed, 18 Mar 2020, Frank Ch. Eigler via Gcc wrote:

> > > The key here is to realize that the raw message is not what you get
> > > back from the mailing list reflector, and also not the raw message
> > > that was sent by the sender.  In this day of mta intermediaries,
> > > proxies, reflectors, it may be time to revisit that suggestion.
> > 
> > But these largely are new problems.  It used to work flawlessly.
> 
> I understand that's frustrating.  But these workflows were counting on
> literally unspecified behaviours not changing, or outright standards
> violations continuing.

Wut?  How is "not mangle the mail body" in any way violating standards?  
You're talking about rewriting or adding headers (where the former is Real 
Bad, no matter what DMARC wants to impose), but the suggestion is based on 
not rewriting the body.  If the body (including attachtments) is rewritten 
any way then that simply is a bug.

> > Patch reencoding problems go back to the redhat.com changes last
> > November (I understand the responsible vendor is working on a fix,
> > but I'm not up-to-date on the current developments).
> 
> This one is a standards-compliant reencoding.  Even if mimecast (?)
> stops doing it, we can't be sure nothing else will.
> 
> > Since the sourceware.org Mailman migration, the From: header is being
> > rewritten, without any compelling reason.  I certainly do not do any
> > DMARC checking here, so the rewriting does not benefit me.
> 
> It benefits you because more and more email services are rejecting or
> interfering with mail that is not clean enough.  If you want to
> receive mail reliably, or send and have confidence that it is
> received, clean mail benefits you.

Depends on your definition of "clean".  If by that you mean rewriting mail 
bodies then I'm not sure what to say.


Ciao,
Michael.

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -

> [...]  You're talking about rewriting or adding headers (where the
> former is Real Bad, no matter what DMARC wants to impose), but the
> suggestion is based on not rewriting the body.  If the body
> (including attachtments) is rewritten any way then that simply is a
> bug. [...]

We're mixing two things.

The From: header rewriting for DMARC participants is something sourceware
is doing now.

The Content-Transfer-Encoding: change is done by intermediate MTA's
whose identity is unknown.  (I don't believe this behaviour is
forbidden by RFCs, but even if it were, we may have no way of fixing
the mystery MTA.)


- FChE

Re: Not usable email content encoding

[Bernd Schmidt]

On 3/18/20 3:22 PM, Frank Ch. Eigler via Gcc wrote:
> The From: header rewriting for DMARC participants is something sourceware
> is doing now.

Out of curiousity, is this rewriting you are talking about the cause for 
a lot of mails showing up as "From: GCC List" rather than their real 
senders? This has become very annoying recently.


Bernd

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -

> > The From: header rewriting for DMARC participants is something sourceware
> > is doing now.
> 
> Out of curiousity, is this rewriting you are talking about the cause for a
> lot of mails showing up as "From: GCC List" rather than their real senders?
> This has become very annoying recently.

Yes, for emails from domains with declared interest in email
cleanliness, via DMARC records in DNS.  We have observed mail
-blocked- at third parties, even just days ago, when we failed to
sufficiently authenticate outgoing reflected emails.

AIUI, all this effort is driven by wanting to defeat not just spammers
but also real security problems like phishing enabled by forgery,
including specifically the From: header.

- FChE

Re: Not usable email content encoding

[Michael Matz]

Hello,

On Wed, 18 Mar 2020, Frank Ch. Eigler via Gcc wrote:

> > > The From: header rewriting for DMARC participants is something sourceware
> > > is doing now.
> > 
> > Out of curiousity, is this rewriting you are talking about the cause for a
> > lot of mails showing up as "From: GCC List" rather than their real senders?
> > This has become very annoying recently.
> 
> Yes, for emails from domains with declared interest in email
> cleanliness, via DMARC records in DNS.  We have observed mail
> -blocked- at third parties, even just days ago, when we failed to
> sufficiently authenticate outgoing reflected emails.

Was this blocking also a problem before mailman (i.e. two weeks ago)?  
Why did nobody scream for not having received mail?  Or why is it blocked 
now, but wasn't before?  Can it be made so again, like it was with ezmlm?

(And DMARCs requirement of having to rewrite From: headers should make it 
clear to everyone that it's stupid).


Ciao,
Michael.

Re: Not usable email content encoding

[Christopher Faylor]

On Wed, Mar 18, 2020 at 04:42:13PM +0000, Michael Matz wrote:
>On Wed, 18 Mar 2020, Frank Ch. Eigler via Gcc wrote:
>> Yes, for emails from domains with declared interest in email
>> cleanliness, via DMARC records in DNS.  We have observed mail
>> -blocked- at third parties, even just days ago, when we failed to
>> sufficiently authenticate outgoing reflected emails.
>
>Was this blocking also a problem before mailman (i.e. two weeks ago)?  
>Why did nobody scream for not having received mail?  Or why is it blocked 
>now, but wasn't before?  Can it be made so again, like it was with ezmlm?

We *were* seeing an increased number of bounces because qmail wasn't
doing the right thing.  People were complaining to postmaster that they
hadn't received email.  I was dreading having to figure out what DMARC
issues they were having with ezmlm/qmail.  So, it's nice for us admin
volunteers to have a standard system to use that doesn't require
searching for patches or workarounds non-standard systems like ezmlm/qmail.

>(And DMARCs requirement of having to rewrite From: headers should make it 
>clear to everyone that it's stupid).

I hate the rewriting quite a bit.  But, if major players like gmail have
adopted DMARC then there really isn't much we can do except to play
along.  It is galling, but the alternative is telling people not to use
gmail or yahoo.  We obviously can't do that.

I'm not sure that everyone understands that on the old server we were
using an ancient, hacked email system of qmail and ezmlm.  I'd added
unofficial (if official even means anything) patches and work arounds
to allow us to continue to communicate with, e.g., yahoo and qmail but
they never worked 100% right and, as mentioned, we were starting to see
bounces and complaints.  Even the old system munged From addresses, too.

We decided for server2 to standardize on postfix + mailman because both
were available on RHEL.  Using postfix + mailman means that if anything
happens to fche or cgf, it should be possible for some hypothetical
volunteer to figure out how email works on sourceware.

There is at least one site that offers qmail and ezmlm packages but
the support seeemed somewhat iffy.  And, if we had used those instead,
and they had updated their support for DMARC/DKIM/SPF then we'd be
in the same boat anyway.

mailman is supposed to only munge From addresses from domains that
enforce DMARC but we're finding that some domains don't properly
advertise themselves and email is not going through to them.  So, it is
looking like we may have to turn it on universally.

The bottom line is that mailman tries harder to make sure that mail gets
through than the kludge I added to qmail ever did so, in theory, more
mail should get through once we finish tweaking.

FWIW, I rewrote my local email delivery agent so that it it puts most
of the From addresses back the way they were before.  It catches most
of the munging so that I don't even notice it anymore.

cgf

Re: Not usable email content encoding

[Jim Wilson]

I'm one of the old timers that likes our current work flow, but even I
think that we are risking our future by staying with antiquated tools.
One of the first things I need to teach new people is now to use email
"properly".  It is a barrier to entry for new contributors, since our
requirements aren't how the rest of the world uses email anymore.
LLVM has phabricator.  Some git based projects are using gerrit.
Github and gitlab are useful services.  We need to think about setting
up easier ways for people to submit patches, rather than trying to fix
all of the MUAs and MTAs in the world.

Jim

Re: Not usable email content encoding

[Jonathan Wakely]

On Wed, 18 Mar 2020 at 21:54, Jim Wilson wrote:
>
> I'm one of the old timers that likes our current work flow, but even I
> think that we are risking our future by staying with antiquated tools.
> One of the first things I need to teach new people is now to use email
> "properly".  It is a barrier to entry for new contributors, since our
> requirements aren't how the rest of the world uses email anymore.
> LLVM has phabricator.

Which is horrible.

> Some git based projects are using gerrit.

Which I looked into previously and decided I didn't like it. If I
recall correctly, gerrit has to "own" the repo, and so it's only
possible to commit to the repo by pushing to gerrit first, then
getting the patch approved. That is fine for write-after-approval, but
adds a step for maintainers who can approve their own changes.

I think it also only very recently gained the ability to group a
series of patches together, as it wants a single commit per review.


> Github and gitlab are useful services.  We need to think about setting
> up easier ways for people to submit patches, rather than trying to fix
> all of the MUAs and MTAs in the world.

There's also https://pagure.io/pagure which is Free Software.

I think it would be great if one of those forge services could be set
up to run *in parallel* to our existing workflow, so that we can
accept pull requests there without forcing all work to go through it
(like gerrit). New contributors who are used to the GitHub model could
submit pull requests there, and maintainers could merge them into the
repo with the click of a button. For significant changes it's more
likely maintainers would pull the branch into their local repo, test
it, and then push manually. But if we had a CI service testing pull
requests (like Travis CI does for GitHub) and a pull request was shown
to introduce no regressions then it could be merged with a single
click.

Re: Not usable email content encoding

[Jonathan Wakely]

N.B. the CC list has got too big and is causing posts to this thread
to be held for moderator approval.

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -

> N.B. the CC list has got too big and is causing posts to this thread
> to be held for moderator approval.

Ah, can cycle through the lists and raise that limit.
The default 10 is too low.

- FChE

Re: Not usable email content encoding

[Christopher Faylor]

On Wed, Mar 18, 2020 at 06:44:15PM -0400, Frank Ch. Eigler wrote:
>> N.B. the CC list has got too big and is causing posts to this thread
>> to be held for moderator approval.
>
>Ah, can cycle through the lists and raise that limit.
>The default 10 is too low.

Didn't you have to lower that limit for outpost, fche?

I believe it used to be 10 for server1 too, though, fwiw.

cgf

Re: Not usable email content encoding

[Christopher Faylor]

On Wed, Mar 18, 2020 at 11:30:22PM -0400, Christopher Faylor wrote:
>On Wed, Mar 18, 2020 at 06:44:15PM -0400, Frank Ch. Eigler wrote:
>>> N.B. the CC list has got too big and is causing posts to this thread
>>> to be held for moderator approval.
>>
>>Ah, can cycle through the lists and raise that limit.
>>The default 10 is too low.
>
>Didn't you have to lower that limit for outpost, fche?
>
>I believe it used to be 10 for server1 too, though, fwiw.

Another annoying thing about mailman is this double inclusion of the
same address in the Cc.

cgf

Re: Not usable email content encoding

[Joseph Myers]

On Wed, 18 Mar 2020, Jonathan Wakely via Gcc wrote:

> > Some git based projects are using gerrit.
> 
> Which I looked into previously and decided I didn't like it. If I
> recall correctly, gerrit has to "own" the repo, and so it's only

The glibc experiment with gerrit worked without it owning the repo.  
There were a few issues with email notifications that were addressed by 
local patches to gerrit, and a few other such issues with email 
interaction that didn't get addressed but looked like they could have been 
- but nothing for which addressing it seemed unacceptable upstream or that 
looked fatal to using gerrit without it owning the repo and (given some 
fixes) working reasonably well with email (similarly well to Bugzilla, say 
- sending email notifications to a list with sensible content, doing 
something sensible with email replies).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Not usable email content encoding

[Jonathan Wakely]

On Wed, 18 Mar 2020 at 22:45, Joseph Myers wrote:
>
> On Wed, 18 Mar 2020, Jonathan Wakely via Gcc wrote:
>
> > > Some git based projects are using gerrit.
> >
> > Which I looked into previously and decided I didn't like it. If I
> > recall correctly, gerrit has to "own" the repo, and so it's only
>
> The glibc experiment with gerrit worked without it owning the repo.

Ah, good to know, thanks.

Re: Not usable email content encoding

[Tom Tromey]

>>>>> "Jonathan" == Jonathan Wakely via Gcc <gcc@gcc.gnu.org> writes:

[gerrit]
Jonathan> I think it also only very recently gained the ability to group a
Jonathan> series of patches together, as it wants a single commit per review.

We tried gerrit for gdb for a while, and in the end decided to drop it.

The main issue for us is that gerrit's support for patch series is poor.
In particular, it doesn't have any way to provide a cover letter (like
git send-email --compose), but in gdb we rely on these to provide an
introduction to the series -- to help justify the series overall and
orient the reviewers.

Here's the gerrit bug:

    https://bugs.chromium.org/p/gerrit/issues/detail?id=924

Based on this I think we all assumed that the situation wouldn't
improve.

Also, gerrit was pretty bad about threading messages, so it became quite
hard to follow progress in email (but following all patches in the web
interface is very difficult, a problem shared by all these web UIs).

Phabricator, IME, is even worse.  Last I used it, it had extremely bad
support for patch series, to the extent that Mozilla had to write a tool
wrapping Phabricator to make it workable.

In gdb we've also considered using an updated patchworks -- with a
gerrit-like commit hook it would be possible to automatically close
patches when they land, which is patchworks' biggest weakness.  (In gdb
land we're more concerned with tracking unreviewed patches than with
online patch review.)  However, this probably would not be a good match
for the new From munging, because it would mean extra (forgettable)
steps when trying to apply patches from the patchworks repository.

TL;DR we're doomed,
Tom

Re: Not usable email content encoding

[Florian Weimer]

* Tom Tromey:

> Also, gerrit was pretty bad about threading messages, so it became quite
> hard to follow progress in email (but following all patches in the web
> interface is very difficult, a problem shared by all these web UIs).

What I found most disappointing was that the web interface doesn't
guide you to the next reasonable step for your reviews and patches,
like showing comments which need addressing.  Tagging messages in an
email client for later action actually works better than that, I think.

Re: Not usable email content encoding

[Richard Biener]

On Thu, Mar 19, 2020 at 2:28 PM Florian Weimer <fw@deneb.enyo.de> wrote:
>
> * Tom Tromey:
>
> > Also, gerrit was pretty bad about threading messages, so it became quite
> > hard to follow progress in email (but following all patches in the web
> > interface is very difficult, a problem shared by all these web UIs).
>
> What I found most disappointing was that the web interface doesn't
> guide you to the next reasonable step for your reviews and patches,
> like showing comments which need addressing.  Tagging messages in an
> email client for later action actually works better than that, I think.

I guess if anything we'd want something git-centric now like github
or gitlab pull requests & reviews.  The only complication is approval
then which would still mean manual steps.  Patch review would also not
be publicly visible and archived(?) so both chiming in late after visible
progress and archeology would be harder.  I think following all
patch reviews by clicking on websites rather than watching gcc-patches
is impractical.

Richard.

Re: Not usable email content encoding

[Florian Weimer]

* Richard Biener:

> On Thu, Mar 19, 2020 at 2:28 PM Florian Weimer <fw@deneb.enyo.de> wrote:
>>
>> * Tom Tromey:
>>
>> > Also, gerrit was pretty bad about threading messages, so it became quite
>> > hard to follow progress in email (but following all patches in the web
>> > interface is very difficult, a problem shared by all these web UIs).
>>
>> What I found most disappointing was that the web interface doesn't
>> guide you to the next reasonable step for your reviews and patches,
>> like showing comments which need addressing.  Tagging messages in an
>> email client for later action actually works better than that, I think.
>
> I guess if anything we'd want something git-centric now like github
> or gitlab pull requests & reviews.  The only complication is approval
> then which would still mean manual steps.

Gitlab has a “merge if CI passes” button, I think.  There are similar
third-party solutions for Github.

> Patch review would also not be publicly visible and archived(?) so
> both chiming in late after visible progress and archeology would be
> harder.

The comments could be archived on a mailing list.  But there is only
some threading.  Maybe this can be compensated to some extent by
producing smaller patches, so that there's less reason for
mega-review-threads.

I don't know if Gitlab supports patch reviews, or if you only can
review all the squashed changes in a merge request.  Github and Gerrit
support comments on individual patches, but Pagure does not, so this
is not something that has universal support in all such tools.  (I
would consider such a feature critical for glibc development.)

I haven't figured out yet under what circumstances Github links
commits to pull requests in the web UI.  I'm not sure if this
relationship is available outside the web UI.  (Obviously, this is
missing from the current gcc-patches feed as well.)

Re: Not usable email content encoding

[Segher Boessenkool]

On Thu, Mar 19, 2020 at 02:58:55PM +0100, Florian Weimer wrote:
> * Richard Biener:
> > I guess if anything we'd want something git-centric now like github
> > or gitlab pull requests & reviews.  The only complication is approval
> > then which would still mean manual steps.
> 
> Gitlab has a “merge if CI passes” button, I think.  There are similar
> third-party solutions for Github.

To actually commit patches is even more trivial than it was with SVN.
This is not the hard step.

"Merge if CI passes"...  Well, it would be nice if we *had* some CI
system :-)  (But commits should *not* depend on that, and conversely,
you should be able to run something on the CI without committing it,
as well.)

> The comments could be archived on a mailing list.  But there is only
> some threading.  Maybe this can be compensated to some extent by
> producing smaller patches, so that there's less reason for
> mega-review-threads.

We often have discussions on gcc-patches@ (instead of on gcc@).  This
is mostly natural, and artificially moving that to another ML because
some tool is lousy is making new problems, not solving any.


Segher

RE: Not usable email content encoding

[Tamar Christina]

> -----Original Message-----
> From: Gcc <gcc-bounces@gcc.gnu.org> On Behalf Of Florian Weimer
> Sent: Thursday, March 19, 2020 1:59 PM
> To: Richard Biener <richard.guenther@gmail.com>
> Cc: overseers@gcc.gnu.org; Jonathan Wakely via Gcc <gcc@gcc.gnu.org>;
> Overseers mailing list <overseers@sourceware.org>; Segher Boessenkool
> <segher@kernel.crashing.org>; Alexander Monakov <amonakov@ispras.ru>;
> Frank Ch. Eigler <fche@elastic.org>; Frank Ch. Eigler <fche@redhat.com>;
> Tom Tromey <tom@tromey.com>
> Subject: Re: Not usable email content encoding
> 
> * Richard Biener:
> 
> > On Thu, Mar 19, 2020 at 2:28 PM Florian Weimer <fw@deneb.enyo.de>
> wrote:
> >>
> >> * Tom Tromey:
> >>
> >> > Also, gerrit was pretty bad about threading messages, so it became
> >> > quite hard to follow progress in email (but following all patches
> >> > in the web interface is very difficult, a problem shared by all these web
> UIs).
> >>
> >> What I found most disappointing was that the web interface doesn't
> >> guide you to the next reasonable step for your reviews and patches,
> >> like showing comments which need addressing.  Tagging messages in an
> >> email client for later action actually works better than that, I think.
> >
> > I guess if anything we'd want something git-centric now like github or
> > gitlab pull requests & reviews.  The only complication is approval
> > then which would still mean manual steps.
> 
> Gitlab has a “merge if CI passes” button, I think.  There are similar third-party
> solutions for Github.

A bit late to the party, but this really doesn't work that well because until recent version of gitlab there was no fairness guarantee.
another patch could be approved after mine (with hours in between because of CI) and yet still get merged first causing my own patch
to no longer apply, you'd rebase and roll the dice again.  To fix this they added merge trains https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/

but trains for GCC Will likely be very short because of Changelog conflicts.
So I don't think an automated merge workflow would work for projects where every single commit changes the same files.

For these workflows to work as well you'd need to have a policy that every single commit go through CI, as a non-green master could flush your entire train if it's rebased on a bad commit.

> 
> > Patch review would also not be publicly visible and archived(?) so
> > both chiming in late after visible progress and archeology would be
> > harder.
> 
> The comments could be archived on a mailing list.  But there is only some
> threading.  Maybe this can be compensated to some extent by producing
> smaller patches, so that there's less reason for mega-review-threads.
> 
> I don't know if Gitlab supports patch reviews, or if you only can review all the
> squashed changes in a merge request.  Github and Gerrit support comments
> on individual patches, but Pagure does not, so this is not something that has
> universal support in all such tools.  (I would consider such a feature critical for
> glibc development.)
> 
> I haven't figured out yet under what circumstances Github links commits to
> pull requests in the web UI.  I'm not sure if this relationship is available
> outside the web UI.  (Obviously, this is missing from the current gcc-patches
> feed as well.)

Re: Not usable email content encoding

[Florian Weimer]

* Tamar Christina:

> A bit late to the party, but this really doesn't work that well
> because until recent version of gitlab there was no fairness
> guarantee.  another patch could be approved after mine (with hours
> in between because of CI) and yet still get merged first causing my
> own patch to no longer apply, you'd rebase and roll the dice again.
> To fix this they added merge trains
> https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/
>
> but trains for GCC Will likely be very short because of Changelog
> conflicts.  So I don't think an automated merge workflow would work
> for projects where every single commit changes the same files.

I had not thought about that.

Does Gitlab support pluggable merge helpers?  The gnulib changelog
auto-merger did a great job when we were still writing changelogs for
glibc.

Re: Not usable email content encoding

[Richard Biener]

On Thu, Apr 23, 2020 at 7:47 AM Florian Weimer <fw@deneb.enyo.de> wrote:
>
> * Tamar Christina:
>
> > A bit late to the party, but this really doesn't work that well
> > because until recent version of gitlab there was no fairness
> > guarantee.  another patch could be approved after mine (with hours
> > in between because of CI) and yet still get merged first causing my
> > own patch to no longer apply, you'd rebase and roll the dice again.
> > To fix this they added merge trains
> > https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/
> >
> > but trains for GCC Will likely be very short because of Changelog
> > conflicts.  So I don't think an automated merge workflow would work
> > for projects where every single commit changes the same files.
>
> I had not thought about that.
>
> Does Gitlab support pluggable merge helpers?  The gnulib changelog
> auto-merger did a great job when we were still writing changelogs for
> glibc.

Btw, I encourage everybody trying to experiment with CI to set it up
for release branches first because of the lower check-in count.

Richard.

RE: Not usable email content encoding

[Tamar Christina]

> -----Original Message-----
> From: Florian Weimer <fw@deneb.enyo.de>
> 
> * Tamar Christina:
> 
> > A bit late to the party, but this really doesn't work that well
> > because until recent version of gitlab there was no fairness
> > guarantee.  another patch could be approved after mine (with hours in
> > between because of CI) and yet still get merged first causing my own
> > patch to no longer apply, you'd rebase and roll the dice again.
> > To fix this they added merge trains
> >
> https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_me
> > rged_results/merge_trains/
> >
> > but trains for GCC Will likely be very short because of Changelog
> > conflicts.  So I don't think an automated merge workflow would work
> > for projects where every single commit changes the same files.
> 
> I had not thought about that.
> 
> Does Gitlab support pluggable merge helpers?  The gnulib changelog auto-
> merger did a great job when we were still writing changelogs for glibc.

I'm not sure and couldn't find a clear answer from a look in the docs..
But I believe it's using git for the rebase so might be able to do it with githooks?

It does have some built in conflicts resolution methods but with some surprising limitations:
---
The file, with conflict markers added, is not over 200 KB in size
---

And also

---
Note: GitLab resolves conflicts by creating a merge commit in the source branch that is not automatically merged into the target branch. This allows the merge commit to be reviewed and tested before the changes are merged, preventing unintended changes entering the target branch without review or breaking the build.
---

So you probably don't want it to be aware of the conflict..

Cheers,
Tamar

Re: Not usable email content encoding

[Jakub Jelinek]

On Thu, Apr 23, 2020 at 09:34:20AM +0100, Jonathan Wakely via Gcc wrote:
> I've been having problems with it recently, e.g.
> https://gcc.gnu.org/g:e76100ced607218a3bf had to fix a changelog,
> because https://gcc.gnu.org/g:d76925e46fad09fc9be67 put my changelog
> entry in the wrong place in gcc/testsuite/Changelog, as a result of a
> rebase using merge-changelog.
> 
> Maybe we should follow glibc and get rid of ChangeLog files before
> trying to use automated CI and Git workflows.

I thought we are leaning towards generating the ChangeLog files from commit
messages rather than getting rid of them, but we need a reliable script to
find the ChangeLog snippet in the commit message and split it into the
separate ChangeLog files and reliable checker that will refuse commits with
badly formatted entries in there, because when it comes from commit
messages, we won't be able to fix mistakes in it later.

	Jakub

Re: Not usable email content encoding

[Jeff Law]

On Thu, 2020-04-23 at 09:34 +0100, Jonathan Wakely via Gcc wrote:
> On Thu, 23 Apr 2020 at 06:49, Florian Weimer wrote:
> > * Tamar Christina:
> > 
> > > A bit late to the party, but this really doesn't work that well
> > > because until recent version of gitlab there was no fairness
> > > guarantee.  another patch could be approved after mine (with hours
> > > in between because of CI) and yet still get merged first causing my
> > > own patch to no longer apply, you'd rebase and roll the dice again.
> > > To fix this they added merge trains
> > > https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/
> > > 
> > > but trains for GCC Will likely be very short because of Changelog
> > > conflicts.  So I don't think an automated merge workflow would work
> > > for projects where every single commit changes the same files.
> > 
> > I had not thought about that.
> > 
> > Does Gitlab support pluggable merge helpers?  The gnulib changelog
> > auto-merger did a great job when we were still writing changelogs for
> > glibc.
> 
> I've been having problems with it recently, e.g.
> https://gcc.gnu.org/g:e76100ced607218a3bf had to fix a changelog,
> because https://gcc.gnu.org/g:d76925e46fad09fc9be67 put my changelog
> entry in the wrong place in gcc/testsuite/Changelog, as a result of a
> rebase using merge-changelog.
> 
> Maybe we should follow glibc and get rid of ChangeLog files before
> trying to use automated CI and Git workflows.
This is precisely why I want to get rid of ChangeLogs and instead generate them
from the VCS as part of the release process.  Ultimately I want to be able to use
workflows where I can push a button on something that's gone through a CI cycle
or "git am" and not have to go back and fix things *by hand*.

What we do with ChangeLogs is (*&@#$*)(*&^ insane from an efficient workflow
standpoint.

jeff

Re: Not usable email content encoding

[Segher Boessenkool]

Hi!

On Thu, Apr 23, 2020 at 12:54:04AM +0000, Tamar Christina wrote:
> but trains for GCC Will likely be very short because of Changelog conflicts.

Why that?  Your patches should *not* touch changelog files, ever.

For CI it is probably easiest if the CI never gets to see the changelog
at all, and that isn't hard to do ;-)


Segher

Re: Not usable email content encoding

[Segher Boessenkool]

On Thu, Apr 23, 2020 at 12:56:20PM +0100, Jonathan Wakely wrote:
> On Thu, 23 Apr 2020 at 12:47, Segher Boessenkool
> <segher@kernel.crashing.org> wrote:
> > On Thu, Apr 23, 2020 at 12:54:04AM +0000, Tamar Christina wrote:
> > > but trains for GCC Will likely be very short because of Changelog conflicts.
> >
> > Why that?  Your patches should *not* touch changelog files, ever.
> >
> > For CI it is probably easiest if the CI never gets to see the changelog
> > at all, and that isn't hard to do ;-)
> 
> For the actual CI testing, yes. But if you want an automated "merge
> after CI testing passes" then what you merge needs to add a ChangeLog
> entry. So either we auto-generate the entry when doing the auto-merge,
> or the ChangeLog entry is already there.

Why would we make it part of the patch for that?  That is backwards :-)

If it isn't part of the patch, no matter how we do it I don't see how
it would cause conflicts?


Segher

Re: Not usable email content encoding

[Jeff Law]

On Thu, 2020-04-23 at 06:46 -0500, Segher Boessenkool wrote:
> Hi!
> 
> On Thu, Apr 23, 2020 at 12:54:04AM +0000, Tamar Christina wrote:
> > but trains for GCC Will likely be very short because of Changelog conflicts.
> 
> Why that?  Your patches should *not* touch changelog files, ever.
> 
> For CI it is probably easiest if the CI never gets to see the changelog
> at all, and that isn't hard to do ;-)
*Nothing* should touch changelog files :-)  They should be generated from the
VCS.  IMHO of course.

That implies that the patch includes a proper ChangeLog entry in the commit
message.  It also implies that if we muck one up, then we live with it rather
than going back and rewriting them.

jeff

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -

> *Nothing* should touch changelog files :-)  They should be generated from the
> VCS.  IMHO of course.

IMHO: the VCS should be the changelog.

- FChE

Re: Not usable email content encoding

[Segher Boessenkool]

Hi!

On Thu, Apr 23, 2020 at 12:57:43PM -0400, Frank Ch. Eigler wrote:
> > *Nothing* should touch changelog files :-)  They should be generated from the
> > VCS.  IMHO of course.
> 
> IMHO: the VCS should be the changelog.

The VCS shows what changed.  The changelog shows what was meant to be
changed.  The difference is very helpful for reviewers, and for
post-mortem purposes, etc.

This can be more than compensated for if submitters would write better
patch descriptions, of course.  But will that happen?


Segher

Re: Not usable email content encoding

[Segher Boessenkool]

On Thu, Apr 23, 2020 at 09:33:47AM -0600, Jeff Law wrote:
> On Thu, 2020-04-23 at 06:46 -0500, Segher Boessenkool wrote:
> > Hi!
> > 
> > On Thu, Apr 23, 2020 at 12:54:04AM +0000, Tamar Christina wrote:
> > > but trains for GCC Will likely be very short because of Changelog conflicts.
> > 
> > Why that?  Your patches should *not* touch changelog files, ever.
> > 
> > For CI it is probably easiest if the CI never gets to see the changelog
> > at all, and that isn't hard to do ;-)
> *Nothing* should touch changelog files :-)  They should be generated from the
> VCS.  IMHO of course.

My point was that this should *never* be part of patches, already.

> That implies that the patch includes a proper ChangeLog entry in the commit
> message.  It also implies that if we muck one up, then we live with it rather
> than going back and rewriting them.

There also is the middle ground where we still have changelog files, but
normal checkins automatically update those from the commit message.
This is an intermediate step, but also makes it possible (and even easy)
to fix up mistakes.


Segher

Re: Not usable email content encoding

[Tom Tromey]

>>>>> "Segher" == Segher Boessenkool <segher@kernel.crashing.org> writes:

Segher> My point was that this should *never* be part of patches, already.

FWIW, I use a few scripts so that I can keep ChangeLogs as files.
That's what I do when working on gdb.

https://github.com/tromey/git-gnu-changelog

This is easier on the whole, IME, because it means there is no extra
manual step before pushing.

Of course, better would be to remove ChangeLogs entirely (including not
putting anything like them into a commit message), because they are
largely not useful and are just make-work.  Again IMNSHO -- I know there
are some folks who read them, but I basically never have since switching
to git.

Tom

Re: Not usable email content encoding

[Thomas Koenig]

> Of course, better would be to remove ChangeLogs entirely (including not
> putting anything like them into a commit message), because they are
> largely not useful and are just make-work.

I disagree. I find them quite useful.

Re: Not usable email content encoding

[Segher Boessenkool]

Hi!

On Fri, Apr 24, 2020 at 05:48:38PM +0200, Thomas Koenig wrote:
> >Of course, better would be to remove ChangeLogs entirely (including not
> >putting anything like them into a commit message), because they are
> >largely not useful and are just make-work.
> 
> I disagree. I find them quite useful.

For what?  And, can that be provided some other way?


Segher

Re: Not usable email content encoding

[Jeff Law]

On Thu, 2020-04-23 at 15:14 -0600, Tom Tromey wrote:
> > > > > > "Segher" == Segher Boessenkool <segher@kernel.crashing.org> writes:
> 
> Segher> My point was that this should *never* be part of patches, already.
> 
> FWIW, I use a few scripts so that I can keep ChangeLogs as files.
> That's what I do when working on gdb.
> 
> https://github.com/tromey/git-gnu-changelog
> 
> This is easier on the whole, IME, because it means there is no extra
> manual step before pushing.
Right.  And that's really my goal here -- eliminate the manual steps.  Ideally I
want to be able to git am; git push on a good patch.  Manual steps for good
patches need to be excised from the workflow.  The ChangeLog file is a major
problem in that regard.



> 
> Of course, better would be to remove ChangeLogs entirely (including not
> putting anything like them into a commit message), because they are
> largely not useful and are just make-work.  Again IMNSHO -- I know there
> are some folks who read them, but I basically never have since switching
> to git.
I read them less and less.  At this point I think I read them to see if a
particular patch in my queue has already been applied.  Otherwise I'm using the
git info.

jeff

Re: Not usable email content encoding

[Segher Boessenkool]

On Fri, May 01, 2020 at 09:51:53AM -0600, Jeff Law wrote:
> On Thu, 2020-04-23 at 15:14 -0600, Tom Tromey wrote:
> > > > > > > "Segher" == Segher Boessenkool <segher@kernel.crashing.org> writes:
> > 
> > Segher> My point was that this should *never* be part of patches, already.
> > 
> > FWIW, I use a few scripts so that I can keep ChangeLogs as files.
> > That's what I do when working on gdb.
> > 
> > https://github.com/tromey/git-gnu-changelog
> > 
> > This is easier on the whole, IME, because it means there is no extra
> > manual step before pushing.
> Right.  And that's really my goal here -- eliminate the manual steps.  Ideally I
> want to be able to git am; git push on a good patch.  Manual steps for good
> patches need to be excised from the workflow.  The ChangeLog file is a major
> problem in that regard.

I still almost always write the changelog not before sending a patch
series to the mailing list.  This is good *anyway*, it isn't a bad idea
to look at your creation from a high level first, then; and it is also
not a bad idea to look at the details, to see if you missed something.
A "self-review", if you want.

This of course also completely side-steps the issues with keeping the
changelog up-to-date that so many people struggle with (that probably
is what made me start doing this, heh).  (I also have the changelogs
in files of course, that way: the email series I send out I keep as
files).

> > Of course, better would be to remove ChangeLogs entirely (including not
> > putting anything like them into a commit message), because they are
> > largely not useful and are just make-work.  Again IMNSHO -- I know there
> > are some folks who read them, but I basically never have since switching
> > to git.
> I read them less and less.  At this point I think I read them to see if a
> particular patch in my queue has already been applied.  Otherwise I'm using the
> git info.

It helps reviewing tremendously.

Having very good patch factoring would help; having very good commit
messages would help.  Together they are much better than a changelog is.
Without either (i.e., *the current status quo*), there is real value
lost.

And making it easier to shove in garbage is not an advantage :-/


Segher

Re: Not usable email content encoding

[Segher Boessenkool]

On Thu, Mar 19, 2020 at 02:41:05PM +0100, Richard Biener wrote:
> I guess if anything we'd want something git-centric now like github
> or gitlab pull requests & reviews.  The only complication is approval
> then which would still mean manual steps.  Patch review would also not
> be publicly visible and archived(?) so both chiming in late after visible
> progress and archeology would be harder.  I think following all
> patch reviews by clicking on websites rather than watching gcc-patches
> is impractical.

patchwork used to work great as an archive (it runs for GCC for almost
ten years now).  It is not meant as a patch review system: instead, it
is meant as something for (kernel) maintainers where they can pick up
patches to apply (which isn't our workflow: people apply their own
patches, in general).  It is also useful to find neglected patches, but
that only can work if someone (or a group of people) does the work for
that.  This doesn't work for us.

*Used to* work great...  With the From: munging, it now has become
quite hard (and very annoying) to navigate, and even just to read :-(


Segher

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi, Jim -

> [gerrit etc.]

Good points.

> [...]  We need to think about setting up easier ways for people to
> submit patches, rather than trying to fix all of the MUAs and MTAs
> in the world.

Another related point.  We are comingling email as a communication
medium AND a commit transport medium.  For the former, as in patch
review / RFC, one may not require a form of the patch that is finally
committable to master, so the exact From: etc. may not matter.  For
the latter, attachments are more bullet-proof.

- FChE

Re: Not usable email content encoding

[Segher Boessenkool]

Hi!

On Wed, Mar 18, 2020 at 06:33:08PM -0400, Frank Ch. Eigler wrote:
> > [...]  We need to think about setting up easier ways for people to
> > submit patches, rather than trying to fix all of the MUAs and MTAs
> > in the world.
> 
> Another related point.  We are comingling email as a communication
> medium AND a commit transport medium.  For the former, as in patch
> review / RFC, one may not require a form of the patch that is finally
> committable to master, so the exact From: etc. may not matter.

But OTOH, it is extremely valuable to review the commit message at the
same time as the patch.  Which we now *can*, for contributors who follow
a more "git-like" workflow.

> For the latter, attachments are more bullet-proof.

Disregarding binary attachments, which are unworkable with many tools
(and which are disallowed on gcc-patches for that reason), is this
really true?  Do some MUAs (or MTAs) mess up only the first body part
they encounter?  Or what?


Segher

Re: Not usable email content encoding

[Segher Boessenkool]

Hi!

On Wed, Mar 18, 2020 at 02:52:14PM -0700, Jim Wilson wrote:
> I'm one of the old timers that likes our current work flow, but even I
> think that we are risking our future by staying with antiquated tools.

It's not ancient tools, it is low-requirement generic tools, and
everyone can use that to build a workflow that works for him/herself.

> One of the first things I need to teach new people is now to use email
> "properly".  It is a barrier to entry for new contributors, since our
> requirements aren't how the rest of the world uses email anymore.

Knowing how to use email properly is a very useful life skill.

A large part of the Free Software (and open source) world still uses
email as primary communication medium, too.

Also you might want to read  https://lwn.net/Articles/702177/  for a
different viewpoint.

> LLVM has phabricator.  Some git based projects are using gerrit.
> Github and gitlab are useful services.  We need to think about setting
> up easier ways for people to submit patches, rather than trying to fix
> all of the MUAs and MTAs in the world.

People should not just send patches: they need to explain why it would
be a good idea to include that patch, which requires a lot of talking
*about* the patch.  Email is a difficult medium for that, but it is
still _much_ better than any of the code review website things, imo.

The key point is that email is completely free-form, I think?


Segher

Re: Not usable email content encoding

[Maciej W. Rozycki]

On Wed, 18 Mar 2020, Frank Ch. Eigler via Gcc wrote:

> > Out of curiousity, is this rewriting you are talking about the cause for a
> > lot of mails showing up as "From: GCC List" rather than their real senders?
> > This has become very annoying recently.
> 
> Yes, for emails from domains with declared interest in email
> cleanliness, via DMARC records in DNS.  We have observed mail
> -blocked- at third parties, even just days ago, when we failed to
> sufficiently authenticate outgoing reflected emails.
> 
> AIUI, all this effort is driven by wanting to defeat not just spammers
> but also real security problems like phishing enabled by forgery,
> including specifically the From: header.

 And it has actually broken GCC's patchwork: 
<http://patchwork.ozlabs.org/project/gcc/list/>, which I used to use to 
track my patches.  Now I cannot do that anymore as patches submitted from 
my WDC address are marked as coming from <gcc-patches@gcc.gnu.org>, which 
obviously means they are not attributed to me.

 I am fairly sure it breaks `git am' too, requiring a `From' override in 
the change description for author attribution in patch application to work 
reliably (I tend to work on my outbox when applying my own patches, so I 
avoid this issue, but I am sure the issue will hit someone sooner or 
later).

 And of course I cannot use the `macro@' pattern anymore to select mailing 
list messages in my inbox that I sent myself.  Frankly it's the least of 
the annoyances, but still, and they all add up.

 This is all silly, requiring the SMTP envelope sender to match the `From' 
header breaks even the most basic e-mail mechanisms like the use of a 
`.forward' file.  Can we please do something about it?  Is functional 
regression the price we absolutely need to pay for progress?

 How come the Linux kernel people who do e-mail patch management on a 
vastly larger scale than we do, both in terms of traffic and the number of 
mailing list subscribers, can get away without all these odd quirks in 
their list server management?  Perhaps it would be good asking them how 
they handle their stuff?

  Maciej

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -


> [...]
>  I am fairly sure it breaks `git am' too, requiring a `From' override in 
> the change description for author attribution in patch application to work 
> reliably (I tend to work on my outbox when applying my own patches, so I 
> avoid this issue, but I am sure the issue will hit someone sooner or 
> later).

That part is at least pretty easy: use
  git format-patch --from "Real Name git <you@email.com>"
which will then force a second fake From: header into the body of
the commit email, where git-am can find it.

> This is all silly, requiring the SMTP envelope sender to match the `From' 
> header breaks even the most basic e-mail mechanisms like the use of a 
> `.forward' file.  [...]

Unfortunately naive .forward based forwarding looks exactly like faked
or spam email to a third party MTAs.


> How come the Linux kernel people who do e-mail patch management on a
> vastly larger scale than we do, both in terms of traffic and the
> number of mailing list subscribers, can get away without all these
> odd quirks in their list server management?  [...]

I'm not sure, but their mails tend to be laden with a vast number of
Cc:'s, which bypass mailing list reflectors.


- FChE

Re: Not usable email content encoding

[Maciej W. Rozycki]

Hi Frank,

> >  I am fairly sure it breaks `git am' too, requiring a `From' override in 
> > the change description for author attribution in patch application to work 
> > reliably (I tend to work on my outbox when applying my own patches, so I 
> > avoid this issue, but I am sure the issue will hit someone sooner or 
> > later).
> 
> That part is at least pretty easy: use
>   git format-patch --from "Real Name git <you@email.com>"
> which will then force a second fake From: header into the body of
> the commit email, where git-am can find it.

 Sure, there are various ways to deal with that, both on the sender's and 
the receiver's side, however all require a manual intervention and are 
therefore prone to a human error.  Which will obviously happen sooner or 
later.

> > This is all silly, requiring the SMTP envelope sender to match the `From' 
> > header breaks even the most basic e-mail mechanisms like the use of a 
> > `.forward' file.  [...]
> 
> Unfortunately naive .forward based forwarding looks exactly like faked
> or spam email to a third party MTAs.

 And can certainly score a positive though not a definite rating in spam 
qualification.  I don't think we ought to encourage bad IT management 
practices by trying to adapt to them too hard and hurting ourselves (our 
workflow) in the process.

> > How come the Linux kernel people who do e-mail patch management on a
> > vastly larger scale than we do, both in terms of traffic and the
> > number of mailing list subscribers, can get away without all these
> > odd quirks in their list server management?  [...]
> 
> I'm not sure, but their mails tend to be laden with a vast number of
> Cc:'s, which bypass mailing list reflectors.

 Surely all the bots receive messages via a mailing list subscription 
though.  Cc's are solely for maintainers, reviewers or other explicitly 
interested parties.  Some maintainers actually require submissions to come 
via the relevant mailing list rather directly to them.  Somehow it works.

 Thank you for your input regardless!

  Maciej

Re: Not usable email content encoding

[Florian Weimer]

* Jonathan Wakely:

> On Mon, 6 Apr 2020 at 23:00, Maciej W. Rozycki via Gcc <gcc@gcc.gnu.org> wrote:
>>  And can certainly score a positive though not a definite rating in spam
>> qualification.  I don't think we ought to encourage bad IT management
>> practices by trying to adapt to them too hard and hurting ourselves (our
>> workflow) in the process.
>
> What you call "bad IT management practices" includes how Gmail works,
> which a HUGE number of people use.
>
> A number of lists I'm on switched to our current style of minging a
> year or two ago, because gmail users were not receiving mail, because
> gmail was rejecting the mail.

Gmail can drop mail for any reason.  It's totally opaque, so it's a
poor benchmark for any mailing list configuration changes because it's
very hard to tell if a particular change is effective or not.

Many mailing lists have *not* made such changes and continue to work
just fine in the face of restrictive DMARC sender policies and
enforcement at the recipient.

In general, mail drop rates due to DMARC seem to increase in these two
cases if the original From: header is preserved:

* The sender (i.e., the domain mentioned in the From: header)
  publishes a restrictive DMARC policy and the mailing list strips the
  DKIM signature.

* The sender signs parts of the message that the mailing list alters,
  and the mailing list does not strip the DKIM signature.

If neither scenario applies, it's safe to pass through the message
without munging.  The mailing list software can detect this and
restricting the From: header munging to those cases.

I doubt Mailman 2.x can do this, so it is simply a poor choice as
mailing list software at this point.

A possible workaround which works in almost all cases would be to make
sure that Mailman performs as little message munging as possible: Do
not strip MIME multiparts, do not rewrite the subject line to add
prefixes, do not rewrite To: and Cc: lines, and so on.  With such a
configuration, the message would have to assert in a DKIM signature
that it does not have a List-Id (or similar) header, but that this is
really a fringe case.

I believe the OpenJDK lists use such a workaround.  Microsoft and
Google participate there (as far as I know, both microsoft.com and
google.com have restrictive sender policies), and their messages are
only dropped if Mailman filters out a text/html multipart, therefore
editing the MIME headers.

Re: Not usable email content encoding

[Mark Wielaard]

Hi,

On Tue, 2020-04-07 at 11:53 +0200, Florian Weimer via Overseers wrote:
> Gmail can drop mail for any reason.  It's totally opaque, so it's a
> poor benchmark for any mailing list configuration changes because it's
> very hard to tell if a particular change is effective or not.
> 
> Many mailing lists have *not* made such changes and continue to work
> just fine in the face of restrictive DMARC sender policies and
> enforcement at the recipient.
> 
> In general, mail drop rates due to DMARC seem to increase in these two
> cases if the original From: header is preserved:
> 
> * The sender (i.e., the domain mentioned in the From: header)
>   publishes a restrictive DMARC policy and the mailing list strips the
>   DKIM signature.
> 
> * The sender signs parts of the message that the mailing list alters,
>   and the mailing list does not strip the DKIM signature.
> 
> If neither scenario applies, it's safe to pass through the message
> without munging.  The mailing list software can detect this and
> restricting the From: header munging to those cases.
> 
> I doubt Mailman 2.x can do this, so it is simply a poor choice as
> mailing list software at this point.

Earlier versions of Mainman2 had some issues which might accidentally
change some headers. But the latest fixes make this possible. It is how
the FSF handles DMARC for various GNU mailinglists (by NOT modifying
the headers and body and passing through the DKIM signatures):
https://lists.gnu.org/archive/html/savannah-hackers-public/2019-06/msg00018.html

Cheers,

Mark

Re: Not usable email content encoding

[Michael Matz]

Hello,

On Wed, 8 Apr 2020, Mark Wielaard wrote:

> On Tue, 2020-04-07 at 11:53 +0200, Florian Weimer via Overseers wrote:
> > Gmail can drop mail for any reason.  It's totally opaque, so it's a
> > poor benchmark for any mailing list configuration changes because it's
> > very hard to tell if a particular change is effective or not.
> > 
> > Many mailing lists have *not* made such changes and continue to work
> > just fine in the face of restrictive DMARC sender policies and
> > enforcement at the recipient.
> > 
> > In general, mail drop rates due to DMARC seem to increase in these two
> > cases if the original From: header is preserved:
> > 
> > * The sender (i.e., the domain mentioned in the From: header)
> >   publishes a restrictive DMARC policy and the mailing list strips the
> >   DKIM signature.
> > 
> > * The sender signs parts of the message that the mailing list alters,
> >   and the mailing list does not strip the DKIM signature.
> > 
> > If neither scenario applies, it's safe to pass through the message
> > without munging.  The mailing list software can detect this and
> > restricting the From: header munging to those cases.
> > 
> > I doubt Mailman 2.x can do this, so it is simply a poor choice as
> > mailing list software at this point.
> 
> Earlier versions of Mainman2 had some issues which might accidentally 
> change some headers. But the latest fixes make this possible. It is how 
> the FSF handles DMARC for various GNU mailinglists (by NOT modifying the 
> headers and body and passing through the DKIM signatures): 
> https://lists.gnu.org/archive/html/savannah-hackers-public/2019-06/msg00018.html

Oh, that would be nice to have at sourceware.org.  Please?  :-)


Ciao,
Michael.

Re: Not usable email content encoding

[Segher Boessenkool]

Hi!

On Wed, Apr 08, 2020 at 01:50:51PM +0000, Michael Matz wrote:
> On Wed, 8 Apr 2020, Mark Wielaard wrote:
> > Earlier versions of Mainman2 had some issues which might accidentally 
> > change some headers. But the latest fixes make this possible. It is how 
> > the FSF handles DMARC for various GNU mailinglists (by NOT modifying the 
> > headers and body and passing through the DKIM signatures): 
> > https://lists.gnu.org/archive/html/savannah-hackers-public/2019-06/msg00018.html
> 
> Oh, that would be nice to have at sourceware.org.  Please?  :-)

Yes, please please please, can we have this?


Segher

Re: Not usable email content encoding

[Christopher Faylor]

On Wed, Apr 08, 2020 at 04:15:27PM -0500, Segher Boessenkool wrote:
>On Wed, Apr 08, 2020 at 01:50:51PM +0000, Michael Matz wrote:
>>On Wed, 8 Apr 2020, Mark Wielaard wrote:
>>>Earlier versions of Mainman2 had some issues which might accidentally
>>>change some headers.  But the latest fixes make this possible.  It is
>>>how the FSF handles DMARC for various GNU mailinglists (by NOT
>>>modifying the headers and body and passing through the DKIM
>>>signatures):
>>>https://lists.gnu.org/archive/html/savannah-hackers-public/2019-06/msg00018.html
>>
>>Oh, that would be nice to have at sourceware.org.  Please?  :-)
>
>Yes, please please please, can we have this?

In case it isn't obvious, we are already running the latest available
version of mailman 2.

Re: Not usable email content encoding

[Michael Matz]

Hello,

On Mon, 13 Apr 2020, Christopher Faylor wrote:

> On Wed, Apr 08, 2020 at 04:15:27PM -0500, Segher Boessenkool wrote:
> >On Wed, Apr 08, 2020 at 01:50:51PM +0000, Michael Matz wrote:
> >>On Wed, 8 Apr 2020, Mark Wielaard wrote:
> >>>Earlier versions of Mainman2 had some issues which might accidentally
> >>>change some headers.  But the latest fixes make this possible.  It is
> >>>how the FSF handles DMARC for various GNU mailinglists (by NOT
> >>>modifying the headers and body and passing through the DKIM
> >>>signatures):
> >>>https://lists.gnu.org/archive/html/savannah-hackers-public/2019-06/msg00018.html
> >>
> >>Oh, that would be nice to have at sourceware.org.  Please?  :-)
> >
> >Yes, please please please, can we have this?
> 
> In case it isn't obvious, we are already running the latest available
> version of mailman 2.

I think that means that dmarc_moderation_action: "Munge From" can simply 
be switched off then (at least I don't see which other headers e.g. gcc@ 
is rewriting that would cause DMARC to scream; and if there are any, then 
it would be better to disable those as well.  Same with any potential 
body rewriting that might still happen).

I would offer help testing that this doesn't cause delivery issues, e.g. 
on some test email list, but it seems none of my domains is DMARC-infected 
:-/


Ciao,
Michael.

P.S: I wonder btw. why the From munging is enabled also for p=none domains 
like redhat.com.  The RFC says this is to be used for gathering DMARC 
feedback, not requiring any specific action for the mail text itself on 
the sender or receiver.  But an answer to this would be moot with the 
above non-munging of From.

Re: Not usable email content encoding

[Christopher Faylor]

On Mon, Apr 13, 2020 at 10:53:06PM +0000, Michael Matz wrote:
>On Mon, 13 Apr 2020, Christopher Faylor wrote:
>>On Wed, Apr 08, 2020 at 04:15:27PM -0500, Segher Boessenkool wrote:
>>>On Wed, Apr 08, 2020 at 01:50:51PM +0000, Michael Matz wrote:
>>>>On Wed, 8 Apr 2020, Mark Wielaard wrote:
>>>>>Earlier versions of Mainman2 had some issues which might accidentally
>>>>>change some headers.  But the latest fixes make this possible.  It is
>>>>>how the FSF handles DMARC for various GNU mailinglists (by NOT
>>>>>modifying the headers and body and passing through the DKIM
>>>>>signatures):
>>>>>https://lists.gnu.org/archive/html/savannah-hackers-public/2019-06/msg00018.html
>>>>
>>>>Oh, that would be nice to have at sourceware.org.  Please?  :-)
>>>
>>>Yes, please please please, can we have this?
>>
>>In case it isn't obvious, we are already running the latest available
>>version of mailman 2.
>
>I think that means that dmarc_moderation_action: "Munge From" can
>simply be switched off then (at least I don't see which other headers
>e.g.  gcc@ is rewriting that would cause DMARC to scream; and if there
>are any, then it would be better to disable those as well.  Same with
>any potential body rewriting that might still happen).

It can't be switched off if we strip html attachments.

>I would offer help testing that this doesn't cause delivery issues, e.g. 
>on some test email list, but it seems none of my domains is DMARC-infected 
>:-/

Nevertheless, offers of help are appreciated (and rare).  :-)

cgf

Re: Not usable email content encoding

[Maciej W. Rozycki]

On Tue, 14 Apr 2020, Christopher Faylor wrote:

> >I think that means that dmarc_moderation_action: "Munge From" can
> >simply be switched off then (at least I don't see which other headers
> >e.g.  gcc@ is rewriting that would cause DMARC to scream; and if there
> >are any, then it would be better to disable those as well.  Same with
> >any potential body rewriting that might still happen).
> 
> It can't be switched off if we strip html attachments.

 Based on current usage would instead plain rejecting e-mail with any HTML 
attachments be a viable way to move forward?

> >I would offer help testing that this doesn't cause delivery issues, e.g. 
> >on some test email list, but it seems none of my domains is DMARC-infected 
> >:-/
> 
> Nevertheless, offers of help are appreciated (and rare).  :-)

 I'm in a `p=none' domain at this e-mail address, so technically I would 
say it has been quite reasonably configured and therefore none of our 
people should have to choose between suffering from `From' munging or 
persuading the postmasters to change the configuration or switching to 
another MTA for mailing list submissions.

 Therefore I'd be happy to participate in testing a solution for disabling 
`From' munging for `p=none' domains.

  Maciej

Re: Not usable email content encoding

[Christopher Faylor]

On Tue, Apr 14, 2020 at 10:38:12PM +0100, Maciej W. Rozycki wrote:
>Therefore I'd be happy to participate in testing a solution for
>disabling `From' munging for `p=none' domains.

I can't speak for fche but I have a limited amount of time for any
computer activity right now.  A pinched nerve in my neck makes typing
difficult.

However, as a first step, please subscribe to test-list@sourceware.org.
If we do any tinkering it will be there.  I'll try to play with settings
in the next few days.

I think we can bounce email which seems to be html.  As I'm sure you all know,
we used to do that on server1.  Then we switched to stripping the html
attachment if there was a text component to the email.  At the time, I don't
think I had the slightest notion that there would be ramifications for DMARC.

Is this something that needs to be voted on?  I don't care either way.  I think
fche would probably not mind letting html through.

Do other mailing lists have policies about this?  Is it a given that all of the
technical mailing lists would be ok with sending text-only?  I know it greatly
confuses cygwin users but many of them are not that technical.

cgf

Re: Not usable email content encoding

[Michael Matz]

Hello,

On Tue, 7 Apr 2020, Jonathan Wakely via Gcc wrote:

> On Mon, 6 Apr 2020 at 23:00, Maciej W. Rozycki via Gcc <gcc@gcc.gnu.org> wrote:
> >  And can certainly score a positive though not a definite rating in spam
> > qualification.  I don't think we ought to encourage bad IT management
> > practices by trying to adapt to them too hard and hurting ourselves (our
> > workflow) in the process.
> 
> What you call "bad IT management practices" includes how Gmail works,
> which a HUGE number of people use.
> 
> A number of lists I'm on switched to our current style of minging a
> year or two ago, because gmail users were not receiving mail, because
> gmail was rejecting the mail.

I find that unconvincing, because even googlegroup email lists don't 
mangle From: from sender domains that are now mangled by sourceware.org 
:-/

Can we please switch it off?  It's not like we really had a problem before 
the switch to mailman.


Ciao,
Michael.

Re: Not usable email content encoding

[Frank Ch. Eigler]

Hi -

> > A number of lists I'm on switched to our current style of minging a
> > year or two ago, because gmail users were not receiving mail, because
> > gmail was rejecting the mail.
> 
> I find that unconvincing, because even googlegroup email lists don't 
> mangle From: from sender domains that are now mangled by sourceware.org 
> :-/

It turns out receiving mail FROM google-groups mail is itself
sometimes at risk because it fails to do this From: mangling, and its
ARC/DKIM re-signature of mail requires even more software to process
and bless.  (Its current behaviour on some groups-gmail lists I'm on
are DMARC non-compliant.)

> Can we please switch it off?  It's not like we really had a problem
> before the switch to mailman.

We have offered some first-hand evidence that there were problems,
just they were worked on by people in the background.


- FChE

Re: Not usable email content encoding

[Michael Matz]

Hello,

On Tue, 7 Apr 2020, Frank Ch. Eigler wrote:

> > I find that unconvincing, because even googlegroup email lists don't 
> > mangle From: from sender domains that are now mangled by sourceware.org 
> > :-/
> 
> It turns out receiving mail FROM google-groups mail is itself
> sometimes at risk because it fails to do this From: mangling, and its
> ARC/DKIM re-signature of mail requires even more software to process
> and bless.  (Its current behaviour on some groups-gmail lists I'm on
> are DMARC non-compliant.)

In a way that's amusing and just reinforces my p.o.v. that DMARC is 
bollocks.

> > Can we please switch it off?  It's not like we really had a problem
> > before the switch to mailman.
> 
> We have offered some first-hand evidence that there were problems,
> just they were worked on by people in the background.

Okay, now the question is, are those problems offsetting the current 
problems?  IMHO they don't, but of course I'm heavily biased, not having 
had those old problems  :)


Ciao,
Michael.

Re: Not usable email content encoding

[Christopher Faylor]

On Tue, Apr 07, 2020 at 03:56:09PM +0000, Michael Matz wrote:
>In a way that's amusing and just reinforces my p.o.v. that DMARC is 
>bollocks.

Not that it means anything but I agree 100%.

It's like whoever made the "standard" just said "to hell with mailing
lists".

Re: Not usable email content encoding

[Maciej W. Rozycki]

On Tue, 7 Apr 2020, Christopher Faylor wrote:

> >In a way that's amusing and just reinforces my p.o.v. that DMARC is 
> >bollocks.
> 
> Not that it means anything but I agree 100%.
> 
> It's like whoever made the "standard" just said "to hell with mailing
> lists".

 Maybe they just didn't know of their existence.  Seriously.

 I have actually been more and more astonished recently with the discovery 
of people living in some kind of a parallel reality of Internet standards 
and practices, who never heard of solutions and habits I have known since 
forever and considered obvious.  Especially as we now have an entirely new 
generation of IT people who never actually saw the Internet like it was 
back in 1990s and earlier on, simply because it was before they were born.

 Consequently things that used to work since forever start breaking and 
people you report such breakage to have no clue as to what you're talking 
about.

  Maciej

Re: Not usable email content encoding

[Florian Weimer]

* Maciej W. Rozycki:

> On Tue, 7 Apr 2020, Christopher Faylor wrote:
>
>> >In a way that's amusing and just reinforces my p.o.v. that DMARC is 
>> >bollocks.
>> 
>> Not that it means anything but I agree 100%.
>> 
>> It's like whoever made the "standard" just said "to hell with mailing
>> lists".
>
>  Maybe they just didn't know of their existence.  Seriously.

I think that's extremely unlikely.  If you do not do header munging in
the mailing list software, things will work.  That suggests to me that
the people who worked on DKIM and DMARC had this use case in mind.

But of course if you add subject prefixes, rewrite attachments (and
thus the message body), or enable the Mailman duplicate suppression
feature (which edits To:/Cc: lists of the messages posted to the
list), then things will break.  But that's not required to run a
mailing list.

Re: Not usable email content encoding

[Christopher Faylor]

On Tue, Apr 07, 2020 at 03:13:53PM +0000, Michael Matz wrote:
>Can we please switch it off?  It's not like we really had a problem before 
>the switch to mailman.

You can't really make statements like this which imply that you are
aware of "problems" on sourceware when you're not a sourceware
administrator.

We actually were munging the From on the old server and we were getting
an increasing number of complaints to postmaster that people weren't
receiving mail.

Re: Not usable email content encoding

[Maciej W. Rozycki]

On Tue, 7 Apr 2020, Christopher Faylor wrote:

> >Can we please switch it off?  It's not like we really had a problem before 
> >the switch to mailman.
> 
> You can't really make statements like this which imply that you are
> aware of "problems" on sourceware when you're not a sourceware
> administrator.
> 
> We actually were munging the From on the old server and we were getting
> an increasing number of complaints to postmaster that people weren't
> receiving mail.

 Hmm, methinks these really ought to be bounced to the postmaster of the 
broken receiving site.  It is not that sourceware has been actually doing 
anything specific to make the recipient's MTA refuse e-mail.  Even if some 
spam prevention has been installed to block some kind of e-mail, whoever 
requested some kind of e-mail traffic need to have a way to whitelist the 
traffic requested, and I believe all list server software provides means 
for the recipient to identify the source, such as a `List-ID' header or 
suchlike.

  Maciej

Re: Not usable email content encoding

[Segher Boessenkool]

Hi!

On Mon, Apr 06, 2020 at 09:58:27PM +0100, Maciej W. Rozycki wrote:
> On Wed, 18 Mar 2020, Frank Ch. Eigler via Gcc wrote:
> > > Out of curiousity, is this rewriting you are talking about the cause for a
> > > lot of mails showing up as "From: GCC List" rather than their real senders?
> > > This has become very annoying recently.
> > 
> > Yes, for emails from domains with declared interest in email
> > cleanliness, via DMARC records in DNS.  We have observed mail
> > -blocked- at third parties, even just days ago, when we failed to
> > sufficiently authenticate outgoing reflected emails.
> > 
> > AIUI, all this effort is driven by wanting to defeat not just spammers
> > but also real security problems like phishing enabled by forgery,
> > including specifically the From: header.
> 
>  And it has actually broken GCC's patchwork: 
> <http://patchwork.ozlabs.org/project/gcc/list/>, which I used to use to 
> track my patches.

Yes, I noticed (I run that patchwork).

> Now I cannot do that anymore as patches submitted from 
> my WDC address are marked as coming from <gcc-patches@gcc.gnu.org>, which 
> obviously means they are not attributed to me.

Yup.

>  I am fairly sure it breaks `git am' too, requiring a `From' override in 
> the change description for author attribution in patch application to work 
> reliably (I tend to work on my outbox when applying my own patches, so I 
> avoid this issue, but I am sure the issue will hit someone sooner or 
> later).

Yup.

>  And of course I cannot use the `macro@' pattern anymore to select mailing 
> list messages in my inbox that I sent myself.  Frankly it's the least of 
> the annoyances, but still, and they all add up.

It would break even the built-in mutt "you sent this yourself" detection.

>  This is all silly, requiring the SMTP envelope sender to match the `From' 
> header breaks even the most basic e-mail mechanisms like the use of a 
> `.forward' file.

This even *never* is true for my mail setups, and never has been, and it
all works fine.

> Can we please do something about it?  Is functional 
> regression the price we absolutely need to pay for progress?
> 
>  How come the Linux kernel people who do e-mail patch management on a 
> vastly larger scale than we do, both in terms of traffic and the number of 
> mailing list subscribers, can get away without all these odd quirks in 
> their list server management?  Perhaps it would be good asking them how 
> they handle their stuff?

If someone cannot connect to them because they have a broken mail setup,
the kernel people say "your problem, get a working mail setup", and that
works fine so far (99.999% of people *can* get working mail, just not
always company mail).


Segher

Re: Not usable email content encoding

[Maciej W. Rozycki]

On Tue, 17 Mar 2020, Segher Boessenkool wrote:

> > I'm surprised it's an issue for you: normally your email client
> > would transform quoted-printable and copying would do the right thing
> > (i.e. select actual patch contents, without whitespace munging).
> > 
> > Are you trying to copy from the raw message representation?
> 
> Everyone trying to work with a patch (instead of just the email) always
> is working with the raw message.  Just  patch < mbox  or  git-am mbox
> for example.

 Well, not everyone obviously as I don't, so please be careful speaking 
quantifiers.

 I apply inline patches with `git am -' by piping messages or attachments 
to the command directly from my e-mail client (with its `Pipe' command), 
so I don't care how any message or attachment has been encoded for 
transport as long as the patch hasn't come out damaged once decoded.

 I'd expect any sane e-mail client to support a similar command if it were 
to be used for software development the way we do it, though I realise 
that regrettably some may not be as capable.

  Maciej