Re: UTF-8 in glibc commit messages

[Mike Frysinger <vapier@gentoo.org>]

On 14 Apr 2021 11:08, Paul Eggert wrote:
> On 4/14/21 8:01 AM, Mike Frysinger wrote:
> > can't we be proactive ?  let's go all-in on UTF-8.
> 
> A problem with "all-in" is that UTF-8 has weird characters that can mess 
> things up. The commit message check was originally put in because 
> someone copy-pasted U+2069 POP DIRECTIONAL ISOLATE into a commit message 
> without realizing it. That invisible character breaks simple searches 
> like 'grep -w'.

arguably seems like a missing feature in grep that the user cannot express
word searches that match graphemes.  but we'll prob get sidetracked into
the weeds with that discussion.

> glibc's current check isn't quite right either, as it allows lines like 
> this:
> 
>      Reviewed-by: Adhemerval Zanella  <adhemerval.zanella@linaro.org>
> 
> in which each "space" is actually U+00A0 NO-BREAK SPACE. Although that's 
> valid ISO-8895-15, U+00A0 is another weird character that we arguably 
> shouldn't allow as it can also mess up searches (it's even blacklisted 
> in URLs by some browsers because of the potential for phishing).
> 
> It'd be better to come up with an exact list of acceptable Unicode 
> characters (probably a set of categories with some exceptions). This 
> would be better than the current approach which is either too-generous 
> or (mostly) too-restrictive. But it'd be some work.

it seems like the concern is over accidental things being copied & pasted
(or generated) in a commit message vs them never being used.  if we tried
to ban e.g. all combining characters, that'd implicitly ban use of many
scripts in the commit message, even if they were being used intentionally.

to that end, rather than try and come up with a sloppy policy that requires
constant care & feeding, why not go with a hook that devs can override ?
so with the commit above where U+00A0 was used by accident, the push would
be rejected with something like:
(i'm sure there's prior art out there we could reuse)
remote: Non-ASCII character found in commit message:
remote: line 1234: Reviewed-by:\u00A0Adhemerval Zanella  <adhemerval.zanella@linaro.org>
remote:                        ^
remote: If this was not a mistake, add "-o bypass-commit-encoding-check" to bypass this check.

that should catch all the accidental usage while still allowing people to
double check and say "i meant to use these".
-mike