Re: [PATCH v9] Locales: Cyrillic -> ASCII transliteration table [BZ #2872]

[Egor Kobylkin <egor@kobylkin.com>]

Hi Rafal,
thanks for putting it into a clear issue statement on SH/Sh problem. I'm
totally with you on this being a good thing to discuss. It is orthogonal
to the tests so let me focus on SH/Sh and System A/B problematic here.

Looks like we have three issues:
1. lack of explicit control which transformation to use (System A or
System B) via //TRANSLIT
2. possibility of collision for System B if used CAP/low transcription
for capital letters
3. Cyrillic 'Ð¥'/'Ñ…' (ha) never transcribes to 'H'/'h' as it should per
System B because it's equivalent 'X'/'x' from System A is always present
and takes precedence.

As a solution shouldn't we only keep System B in a new file
transcribe_cyrillic and put it in place as the explicit ASCII
transcription for targeted locales (as opposed to transliteration)?

We would keep System A as translit_cyrillic but won't include it into
this patch. Once you have resolved an issue of having two conflicting
rule-sets but only one key //TRANSLIT you could add the System A back.

The SH/Sh can be decided on either way - seems like an easy change any way.

Please see more discussion on your excellent points below:

On 16.11.18 23:17, Rafal Luzynski wrote:

> Egor, while at this I was thinking about your idea to transliterate
> letters like "Ш" (uppercase) to "SH" (always uppercase) in order to
> distinguish between "Шема" (-> "SHema") and "Схема" (-> "Shema" or
> "Sxema").

to clarify, this SH/Sh collision issue relates only to iconv -f UTF-8 -t
ASCII//TRANSLIT (i.e. System B transcription).
But it's not only SH/Sh, there are following combinations used to
transcribe capital letters:

YO, DJ, YE, TSH, DH, ZH, CZ, CH, SH, SHH, YU, YA, FH, YH, GH, NG, TCZ

Arguably any of them (if not in that CAP/CAP form) could collide with
their CAP/low equivalent from a different word. (there may be language
grammar rules that in fact prevent some but we don't know for sure)

With transcription we are basically striping information from the data,
mapping it into a smaller character set. The idea to keep them in
CAP/CAP is to try to preserve as much information as possible.


> Also you include a rule to transliterate "Ð¥" to "H" or "X" depending
> on which destination characters are available, which I told you
> already that will not work because both "H" and "X" are always
> available and therefore only the first rule will always be used.

Just to have this here for reference, the idea was to have both rules in
one file so

iconv -f UTF-8 -t ASCII//TRANSLIT
will produce ASCII compatible _transcription_ (System B)

iconv -f UTF-8 -t ISO-8859-15//TRANSLIT |
iconv -f ISO-8859-15 -t UTF-8
will produce Latin _transliteration_ as per ISO 9.1995. (System A)

So in fact we have two rules for each letter in the same file (System A
and System B), where System A takes precedence.

I have a question then: isn't this more like a hack than a right thing
to do?

Shouldn't we have two explicit rules for transcription and
transliteration not dependent on a destination character set?


> I still don't like the idea to
> put two uppercase letters in a beginning of a word in titlecase only
> to indicate that there was originally a single letter.  What if we:
> 
> * drop the rule of transliterating "Ð¥" to "H" and transliterate
> always to "X",
This would contradict ISO 9.1995. (System A).
System A was added on Marko's request (so setting him on TO:) I am
neutral on keeping it or dropping it, just to be clear.

> * transliterate uppercase "Ш" to "Sh" (so it will work fine for
> titlecase words)?
> 
> As a result the Latin letter "h" will only appear as part of a
> digraph and never as a transliteration of "Ð¥" and therefore will
> never cause a conflict. Examples:
> 
> * "Шема" -> "Shema", * "Схема" -> "Sxema".
> 
> Will this solve the problem?
This particular rule with h/x would make sense it's own.
But again - it would contradict the standards.
On the other hand, for my personal needs I care less about standards but
about current functionality and data loss because of missing
transcription altogether due to the BZ #2872.

Bests,
Egor