Re: Help needed reviewing Cyrillic -> ASCII transliteration [BZ #2872]

[Siddhesh Poyarekar <siddhesh@gotplt.org>]

On 03/01/19 4:52 PM, Egor Kobylkin wrote:
> Is there a specific way you measure the bloat of the C locale?
> Is it the size of the resulting libc.so.6 file we are concerned with?

I believe it's built into libc.so, so I suppose you'd have to look at 
its file size.

> In terms of the source code we are just adding as many lines as there
> are letters (169 insertions for Cyrillic in this patch v12)

Yeah, it will likely not be much for a single locale, but it may add up 
across locales.  I have no idea how much, it may well be insignificant.

> Just for clarification, the whole point (at least for me) for this patch
> is to have the transliteration when other methods are not available. Or
> when existing programs/systems can not make use of them. The most basic
> example: filenames in Cyrillic on a NAS that get converted to
> ????????.??? and get overwritten in the worst case. So the most value is
> when it works out of box with the C builtin. Other locales can actually
> implement their own variant and explicitly use it if they need one; some
> already have, others may be just fine with the builtin C.

That's a fair point but given the approximation, that specific use case 
may still be flaky.

>>> Additionally we have a disagreement about how should we handle the
>>>  case when a single original uppercase character transliterates
>>> into a digraph in ASCII.  Should both ASCII characters be
>>> uppercase (which is good for all uppercase strings and also good to
>>> emphasize that the original character was single rather than two
>>> separate characters which accidentally transliterate into two
>>> characters making a digraph) or should only the first ASCII
>>> character be uppercase (which is good for the titlecase words which
>>> is common in natural texts)? An example is "Ш" - should it be "SH"
>>> or "Sh"? Note that "Сх" may also produce "Sh" ("S" + "h" -> "Sh").
>>
>>
>> Is that important?
> 
> As in the above example about the files, you would probably agree that
> it's better not to knowingly introduce a failure vector for such basic
> OS operations like working with files. The transliteration
> capitalization collisions have this negative potential. The users that
> need a different specific capitalization can still implement that in
> their locale.

OK, I can see reason for reducing collisions but again, it remains 
flaky.  We could in the interest of moving forward, strive towards 
making it less flaky but at the same time be aware that there may 
eventually be collisions.

Siddhesh