Stripping multipart/alternative messages and accepting the text/plain part.

Stripping multipart/alternative messages and accepting the text/plain part.

[Carlos O'Donell]

Dear Overseers,

I know that @sourceware.org doesn't support receiving
HTML email to the mailing lists. I have read the reasons
in the FAQ.

It would seem that ezmlm can strip out MIME
parts from multipart MIME messages. Some mail
clients use `multipart/alternative' to send both
plain text and html versions of the same message.
On many lists this is considered annoying and
wasteful (messages are usually 2-3 times as large).
You can have ezmlm remove the html part from such
messages by adding the line ``text/html'' to
DIR/mimeremove.

Currently I'm told that the mailing lists are rejecting
such multipart/alternative emails entirely. This is
sad because we are loosing volunteers because of it.

We should be supporting such emails by stripping
the text/html via the ezmlm configure option.

Doing so would allow us to open our mailing lists to
an even wider audience.

Is this possible?

Cheers,
Carlos.

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Per Bothner]

On 02/27/2012 12:52 PM, Carlos O'Donell wrote:
> Dear Overseers,
>
> I know that @sourceware.org doesn't support receiving
> HTML email to the mailing lists. I have read the reasons
> in the FAQ.

Yeah - the linked-to page [0] gives good reasons for not allowing
arbitrary HTML in email.  However. HTML has many benefits [1],
and if you restricting yourself to a safe-and-sane" subset of HTML [2]
then I don't think the reasons in [0] are valid [3].

This is similar to blog sites that allow comments in a restricted
subset of HTML.  With just a Small Matter of Programming [4], we
should allow the same.

[0] http://www.frontierfleet.net/email/

[1] Re-flow, section headers, unambiguous hyper-links, lists/tables.
optionally images, em/strong/code, math, etc etc.

[3] Is there some existing suitable standard for "safe-and-sane" HTML
for emails (or blog comments)?  I'm guessing there is.

[4] (a) The concern about bad font sizes doesn't apply to restricted
HTML - and regardless, the mail reader or user could override that.
(b) The concern about a text mode HTML reader displaying raw tags
seems pretty silly.  Haven't they heard of lynx/elinks/...?
Processing restricted HTML isn't that difficult.
"Many people still use terminals instead of computers to read email."
Name two.
(c) Digest creation is a bogus concern.  The digester can run
the HTML though lynx before creating the digest - if anyone cares.
(Are digests good for much besides having people incorrectly reply
to the digest?  People who are competent to use digests correctly
are competent to set up filters.)
(d) Message/disk size for restricted HTML is trivial.
(e) Viruses are not an issue for restricted HTML.

[4] I realize nobody has spare time, so I'm not complaining.  But this
luddite "HTML email is evil" mantra gets tiresome.  It needs to stop.
-- 
	--Per Bothner
per@bothner.com   http://per.bothner.com/

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Carlos O'Donell]

On Mon, Feb 27, 2012 at 7:46 PM, Per Bothner <per@bothner.com> wrote:
> On 02/27/2012 12:52 PM, Carlos O'Donell wrote:
>>
>> Dear Overseers,
>>
>> I know that @sourceware.org doesn't support receiving
>> HTML email to the mailing lists. I have read the reasons
>> in the FAQ.
>
> Yeah - the linked-to page [0] gives good reasons for not allowing
> arbitrary HTML in email.  However. HTML has many benefits [1],
> and if you restricting yourself to a safe-and-sane" subset of HTML [2]
> then I don't think the reasons in [0] are valid [3].
>
> This is similar to blog sites that allow comments in a restricted
> subset of HTML.  With just a Small Matter of Programming [4], we
> should allow the same.

Though I agree with many of your points,
my goal today is not to tackle such a broad problem.

My goal today is to see if we can enable a volunteer
to contribute to a software project whose mailing
list is hosted on sourceware.org. This particular
volunteer sends multipart/alternative messages which
include both text/html and test/plain parts.

In summary:
===========

Can the mailing list manager, ezmlm-idx, be configured
to accept multipart/alternative message formats and
strip out the text/html parts while allowing through
the text/plain parts?

e.g. Use DIR/mimeremove?
http://www.de.ezmlm.org/faq/FAQ-9.html#mimeremove

This would make posting to the mailing list "Just work"
for mail clients that produce multipart/alternative
messages like those messages product by GMail in rich
text mode.

Cheers,
Carlos.

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Jonathan Larmour]

On 28/02/12 00:46, Per Bothner wrote:
> Yeah - the linked-to page [0] gives good reasons for not allowing
> arbitrary HTML in email.  However. HTML has many benefits [1],
> and if you restricting yourself to a safe-and-sane" subset of HTML [2]
> then I don't think the reasons in [0] are valid [3].
> 
> This is similar to blog sites that allow comments in a restricted
> subset of HTML.  With just a Small Matter of Programming [4], we
> should allow the same.
> 
> [0] http://www.frontierfleet.net/email/
> 
> [1] Re-flow, section headers, unambiguous hyper-links, lists/tables.
> optionally images, em/strong/code, math, etc etc.
> 
> [3] Is there some existing suitable standard for "safe-and-sane" HTML
> for emails (or blog comments)?  I'm guessing there is.

I've never heard of any such standard, but more importantly I don't think
you'll find any HTML-generating mail clients which restrict themselves to
using this subset, so even if such a standard exists, it won't help.

If we just imposed our own rules on which HTML tags are allowable, this
would be a maintenance headache... not just trying to keep up with what
clients can generate, but also the endless users who would get confused
and complain why one message with certain characteristics works, and
another fails. Bear in mind that most of those using HTML clients have no
way to control the actual HTML produced.

> [4] (a) The concern about bad font sizes doesn't apply to restricted
> HTML - and regardless, the mail reader or user could override that.
> (b) The concern about a text mode HTML reader displaying raw tags
> seems pretty silly.  Haven't they heard of lynx/elinks/...?
> Processing restricted HTML isn't that difficult.
> "Many people still use terminals instead of computers to read email."
> Name two.

Perhaps more importantly, we have publicly viewable mailing list archives
for all lists. It would be very challenging to allow those to continue to
be valuable resources while also keeping them both legible and, most
importantly of all, safe. There are already enough spammers and virus
writers exploiting loopholes, and we would get into a lot of trouble if we
end up facilitating that at any point.

> (c) Digest creation is a bogus concern.  The digester can run
> the HTML though lynx before creating the digest - if anyone cares.
> (Are digests good for much besides having people incorrectly reply
> to the digest?  People who are competent to use digests correctly
> are competent to set up filters.)

I know from my project that there are quite a lot of people who only sign
up to digest versions (never understood it myself, but they're there).

If the whole point is to allow the use of HTML features as you mentioned
in your point [1], then isn't the corollary that stripping these "rich
text" features out is likely to cause problems if people are expecting
them to be respected? Markup needed for context will be missing, images
will be stripped, layout will be mangled, and so on.

> [4] I realize nobody has spare time, so I'm not complaining.  But this
> luddite "HTML email is evil" mantra gets tiresome.  It needs to stop.

I'm sure there could be technical solutions to all the above, but it would
require a lot of work.

I think for what you want, what you should be asking for is for each
project to move to web forums instead of email. I think that would be a
hard sell.

Jifl
-- 
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Carlos O'Donell]

On Mon, Feb 27, 2012 at 3:52 PM, Carlos O'Donell
<carlos@systemhalted.org> wrote:
> Dear Overseers,
>
> I know that @sourceware.org doesn't support receiving
> HTML email to the mailing lists. I have read the reasons
> in the FAQ.
>
> It would seem that ezmlm can strip out MIME
> parts from multipart MIME messages. Some mail
> clients use `multipart/alternative' to send both
> plain text and html versions of the same message.
> On many lists this is considered annoying and
> wasteful (messages are usually 2-3 times as large).
> You can have ezmlm remove the html part from such
> messages by adding the line ``text/html'' to
> DIR/mimeremove.
>
> Currently I'm told that the mailing lists are rejecting
> such multipart/alternative emails entirely. This is
> sad because we are loosing volunteers because of it.
>
> We should be supporting such emails by stripping
> the text/html via the ezmlm configure option.
>
> Doing so would allow us to open our mailing lists to
> an even wider audience.
>
> Is this possible?

Has any more thought been given to this topic?

I would like to see progress on this issue since
it impacts the participation of new volunteers
in our community.

Comments?

Cheers,
Carlos.

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Christopher Faylor]

On Sun, Mar 11, 2012 at 11:46:30AM -0400, Carlos O'Donell wrote:
>On Mon, Feb 27, 2012 at 3:52 PM, Carlos O'Donell wrote:
>>Dear Overseers,
>>
>>I know that @sourceware.org doesn't support receiving HTML email to the
>>mailing lists.  I have read the reasons in the FAQ.
>>
>>It would seem that ezmlm can strip out MIME parts from multipart MIME
>>messages.  Some mail clients use `multipart/alternative' to send both
>>plain text and html versions of the same message.  On many lists this
>>is considered annoying and wasteful (messages are usually 2-3 times as
>>large).  You can have ezmlm remove the html part from such messages by
>>adding the line ``text/html'' to DIR/mimeremove.
>>
>>Currently I'm told that the mailing lists are rejecting such
>>multipart/alternative emails entirely.  This is sad because we are
>>loosing volunteers because of it.
>>
>>We should be supporting such emails by stripping the text/html via the
>>ezmlm configure option.
>>
>>Doing so would allow us to open our mailing lists to an even wider
>>audience.
>>
>>Is this possible?
>
>Has any more thought been given to this topic?
>
>I would like to see progress on this issue since it impacts the
>participation of new volunteers in our community.
>
>Comments?

FWIW, I'm not really too keen on modifying the ezmlm configuration for
this.  I don't know what the side effects will be and I'm not really
interested in supporting the inevitable end-user queries when things
don't work exactly as they expect.

I'm comfortable with telling people "no html".  I'm not comfortable with
being expected to debug why their html attachment (which they think
should be fine) is being bounced.

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Carlos O'Donell]

On Sun, Mar 11, 2012 at 2:32 PM, Christopher Faylor
<cgf-use-the-mailinglist-please@sourceware.org> wrote:
>>Has any more thought been given to this topic?
>>
>>I would like to see progress on this issue since it impacts the
>>participation of new volunteers in our community.
>>
>>Comments?
>
> FWIW, I'm not really too keen on modifying the ezmlm configuration for
> this.  I don't know what the side effects will be and I'm not really
> interested in supporting the inevitable end-user queries when things
> don't work exactly as they expect.

OK, has anyone given thought to allowing HTML email?

Such a configuration is certainly simpler than
changing the DIR/mimeremove option.

> I'm comfortable with telling people "no html".  I'm not comfortable with
> being expected to debug why their html attachment (which they think
> should be fine) is being bounced.

That speaks for just allowing HTML email like other
lists do, that way you don't have to debug anything.

Comments on allowing HTML email?

Cheers,
Carlos.

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Jonathan Larmour]

On 11/03/12 22:10, Carlos O'Donell wrote:
> 
> Comments on allowing HTML email?

I thought I, for one, tackled the main issues as part of here:

http://sourceware.org/ml/overseers/2012-q1/msg00070.html

In summary, I think there are a bunch of things that would need to be
resolved first. In summary the most important aspects are:
- not being a conduit for viruses, nor spam (and particularly if images
are embedded into or linked from the mail).
- Keeping the mailing list archives universally viewable (and again it's
vital this is not full of viruses, spam, etc.).
- Keeping mailing list digests working

I think there has to be a solution to all the above issues before
considering enabling HTML mail. Otherwise sourceware will be blacklisted
by RBLs for mail and banned by web content filters on PCs in next to no
time at all.

Jifl

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Carlos O'Donell]

On Sun, Mar 11, 2012 at 7:49 PM, Jonathan Larmour <jifl@jifvik.org> wrote:
> On 11/03/12 22:10, Carlos O'Donell wrote:
>>
>> Comments on allowing HTML email?
>
> I thought I, for one, tackled the main issues as part of here:
>
> http://sourceware.org/ml/overseers/2012-q1/msg00070.html

I am not on the overseers mailing list and your reply did not include
my email (and I did not consider that I should be reading the list).
Thank you for following up with a summary.

Could you please comment on the strategy of using DIR/mimeremove to
allow legitimate use of multipart/alternative email that has a minimum
of a text/plain part?

Think of text/plain as a "safe" rendering of the text/html part that
users send and therefore acceptable to the list as a whole, while
still keeping out the spammers that send only html email.

So far I see the following consensus:

* Per Bothner - General agreement that HTML email has some benefit. No
direct comment on DIR/mimeremove.

* Christopher Faylor - Not keen on DIR/mimeremove for reasons of
stripping html attachments, but not explicitly against the idea.

* Frank Ch. Eigler - Considers DIR/mimeremove sensible.

* Jonathan Larmour - No comment on DIR/mimeremove.

I would like to reach the conclusion that either:

(a) The lists will *not* support multipart/alternative text/plain, and why.

(b) The lists *will* accept only the text/plain portion of a
multipart/alternative.

Knowing either (a) or (b) will allow me to communicate expectations to
our volunteers and allow them to take action accordingly.

I started this conversation with the hope that we might agree to (b)
and I still consider it a valid outcome.

Cheers,
Carlos.

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Frank Ch. Eigler]

Hi, Carlos -

> [...]
> I would like to reach the conclusion that either:
> 
> (a) The lists will *not* support multipart/alternative text/plain, and why.
> 
> (b) The lists *will* accept only the text/plain portion of a
> multipart/alternative.

I'd be fine with (b).  Note that you'll likely need the mimekeep
rather than mimeremove, so as to preclude not just text/html but
image/jpg etc.

- FChE

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Jonathan Larmour]

On 14/03/12 14:59, Frank Ch. Eigler wrote:
> Hi, Carlos -
> 
>> [...]
>> I would like to reach the conclusion that either:
>>
>> (a) The lists will *not* support multipart/alternative text/plain, and why.
>>
>> (b) The lists *will* accept only the text/plain portion of a
>> multipart/alternative.
> 
> I'd be fine with (b).  Note that you'll likely need the mimekeep
> rather than mimeremove, so as to preclude not just text/html but
> image/jpg etc.

I have no problems with that in principle. However we also have to make
sure that legitimate attachments can continue to work, especially patches.
Based on mime types I've seen used over time, that would mean we should
also permit:

text/x-patch
text/x-diff
application/x-patch
application/octet-stream

Sometimes, collections of files are also sent as zip files, with type
application/zip or application/x-zip (I note that
application/x-zip-compressed is currently rejected in mimereject but not
the other two forms).

Are there any other mime types we'd need to consider permitting?

Jifl
-- 
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Frank Ch. Eigler]

Hi -

jifl wrote:

> [...]
> Based on mime types I've seen used over time, that would mean we should
> also permit:
> 
> text/x-patch
> text/x-diff
> application/x-patch
> application/octet-stream

I wouldn't favour /octet-stream, nor archive forms /*zip, which are
often used to carry spam / malware.


- FChE

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Jonathan Larmour]

On 14/03/12 15:26, Frank Ch. Eigler wrote:
> Hi -
> 
> jifl wrote:
> 
>> [...]
>> Based on mime types I've seen used over time, that would mean we should
>> also permit:
>>
>> text/x-patch
>> text/x-diff
>> application/x-patch
>> application/octet-stream
> 
> I wouldn't favour /octet-stream, nor archive forms /*zip, which are
> often used to carry spam / malware.

Unfortunately I have seen octet-stream used for patches given there is no
official type for patch files. It doesn't appear that we reject
octet-stream at present. I do in fact now see we do reject .zip files,
from /qmail/lists/global/mimereject_fns. So I guess it would be fine to
leave them out.

But I see Chris' mail now so perhaps this doesn't matter any more.

Jifl
-- 
--["No sense being pessimistic, it wouldn't work anyway"]-- Opinions==mine

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Christopher Faylor]

On Tue, Mar 13, 2012 at 06:58:35PM -0400, Carlos O'Donell wrote:
>On Sun, Mar 11, 2012 at 7:49 PM, Jonathan Larmour <jifl@jifvik.org> wrote:
>> On 11/03/12 22:10, Carlos O'Donell wrote:
>>>
>>> Comments on allowing HTML email?
>>
>> I thought I, for one, tackled the main issues as part of here:
>>
>> http://sourceware.org/ml/overseers/2012-q1/msg00070.html
>
>I am not on the overseers mailing list and your reply did not include
>my email (and I did not consider that I should be reading the list).
>Thank you for following up with a summary.
>
>Could you please comment on the strategy of using DIR/mimeremove to
>allow legitimate use of multipart/alternative email that has a minimum
>of a text/plain part?
>
>Think of text/plain as a "safe" rendering of the text/html part that
>users send and therefore acceptable to the list as a whole, while
>still keeping out the spammers that send only html email.
>
>So far I see the following consensus:
>
>* Per Bothner - General agreement that HTML email has some benefit. No
>direct comment on DIR/mimeremove.
>
>* Christopher Faylor - Not keen on DIR/mimeremove for reasons of
>stripping html attachments, but not explicitly against the idea.

If that is what you got from my message then I really wasn't clear.

I'll try to be clearer: I'm not interested in dealing with any fallout
from this type of change.  I think it will potentially make supporting
mailing list problems more difficult since it might require debugging of
mail messages to figure out why something did or didn't get through.
So, I think it is an unnecessary complication and, if implemented, I'll
punt any support requests to somebody else.

And, in case it isn't clear, I don't see anyone else (except very
occasionally fche) dealing with mailing list complaints so it would be a
new role for someone to have to handle these issues.  I have been
dealing with postmaster mail for years so someone else would have to
also start actively monitoring it.

I've been biting my tongue but I guess I'll say this too: I have to
wonder about the quality and dedication of any would-be contributor who
is thrown by the requirement of performing a minor tweak to their email
client.  I could see this being an obstacle on, say, the cygwin list
(and it surely is there) but I don't understand why a technical
contributor would find this a challenge.  Or, if it isn't a challenge,
then I don't understand why it would be such a bone of contention that
it would turn them off from ever contributing.

cgf

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Carlos O'Donell]

On Wed, Mar 14, 2012 at 11:33 AM, Christopher Faylor
<cgf-use-the-mailinglist-please@sourceware.org> wrote:
> If that is what you got from my message then I really wasn't clear.

No problem, that's the purpose of good 3-way communication.

> I'll try to be clearer: I'm not interested in dealing with any fallout
> from this type of change.  I think it will potentially make supporting
> mailing list problems more difficult since it might require debugging of
> mail messages to figure out why something did or didn't get through.
> So, I think it is an unnecessary complication and, if implemented, I'll
> punt any support requests to somebody else.
>
> And, in case it isn't clear, I don't see anyone else (except very
> occasionally fche) dealing with mailing list complaints so it would be a
> new role for someone to have to handle these issues.  I have been
> dealing with postmaster mail for years so someone else would have to
> also start actively monitoring it.

Thank you for clarifying your position.

In summary:

* There is loose agreement that it would be a good thing to
  use DIR/mimekeep to allow users with multipart/alternative
  email to participate in mailing list discussions as long as
  they had MIME parts that matched the parts allowed on the
  list. After all, more project volunteers is a good thing.

* Unfortunately many of the mime parts we want to reject are
  also the default parts currently in use for sending
  patches, diffs, etc. Therefore if we don't outright reject
  emails with text/html parts then the application/octect-stream
  part (which we have to allow for Base64 encoded attached patches)
  of the spammers message will arrive on the list and that is
  not acceptable.

* Unfortunately the use of DIR/mimekeep and really any system
  that strips MIME parts would cause confusion amongst users
  and result in additional and unacceptable support load
  for postmaster (given the current volunteers).

It would seem to me that the position of overseers is
that the request to use DIR/mimekeep to enable posting of
multipart/alternative email with text/plain parts carries
with it an unacceptably high price.

Does that sound about right?

Cheers,
Carlos.

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Carlos O'Donell]

On Wed, Mar 14, 2012 at 6:44 PM, Carlos O'Donell
<carlos@systemhalted.org> wrote:
> In summary:
>
> * There is loose agreement that it would be a good thing to
>  use DIR/mimekeep to allow users with multipart/alternative
>  email to participate in mailing list discussions as long as
>  they had MIME parts that matched the parts allowed on the
>  list. After all, more project volunteers is a good thing.
>
> * Unfortunately many of the mime parts we want to reject are
>  also the default parts currently in use for sending
>  patches, diffs, etc. Therefore if we don't outright reject
>  emails with text/html parts then the application/octect-stream
>  part (which we have to allow for Base64 encoded attached patches)
>  of the spammers message will arrive on the list and that is
>  not acceptable.
>
> * Unfortunately the use of DIR/mimekeep and really any system
>  that strips MIME parts would cause confusion amongst users
>  and result in additional and unacceptable support load
>  for postmaster (given the current volunteers).
>
> It would seem to me that the position of overseers is
> that the request to use DIR/mimekeep to enable posting of
> multipart/alternative email with text/plain parts carries
> with it an unacceptably high price.
>
> Does that sound about right?

Ping?

Agreement with the summary?

Disagreement with the summary?

I'd like to have a clear answer to give to my volunteers.

Cheers,
Carlos.

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Jonathan Larmour]

On 22/03/12 04:24, Carlos O'Donell wrote:
> On Wed, Mar 14, 2012 at 6:44 PM, Carlos O'Donell
[ker-snip more rationale]
>> It would seem to me that the position of overseers is
>> that the request to use DIR/mimekeep to enable posting of
>> multipart/alternative email with text/plain parts carries
>> with it an unacceptably high price.
>>
>> Does that sound about right?
> Agreement with the summary?
> 
> Disagreement with the summary?

The summary fits the rationale given, as far as I'm concerned, although
I'm not the only one here obviously. Call it a silent consensus.

Jifl

Re: Stripping multipart/alternative messages and accepting the text/plain part.

[Frank Ch. Eigler]

Hi -

On Sun, Mar 11, 2012 at 06:10:09PM -0400, Carlos O'Donell wrote:
> [...]
> OK, has anyone given thought to allowing HTML email?

Yeah, we're not enthusiastic.

> Such a configuration is certainly simpler than changing the
> DIR/mimeremove option. [...]

This would make much more sense to me.  Or even use DIR/mimekeep
whitelisting for text/* parts.


- FChE