From: Carlos O'Donell <codonell@redhat.com>
To: Rafal Luzynski <digitalfreak@lingonborough.com>,
Marko Myllynen <myllynen@redhat.com>,
"Diego (Egor) Kobylkin" <egor@kobylkin.com>,
"libc-alpha@sourceware.org" <libc-alpha@sourceware.org>,
"libc-locales@sourceware.org" <libc-locales@sourceware.org>,
Siddhesh Poyarekar <siddhesh@gotplt.org>
Cc: Mike Fabian <mfabian@redhat.com>
Subject: Re: [PING^8][PATCH v12] Locales: Cyrillic -> ASCII transliteration [BZ #2872]
Date: Fri, 07 Jun 2019 12:36:00 -0000 [thread overview]
Message-ID: <761147fe-75d8-fbbf-b75a-1b58323254f9@redhat.com> (raw)
In-Reply-To: <956159024.1658672.1559904734686@poczta.nazwa.pl>
On 6/7/19 6:52 AM, Rafal Luzynski wrote:
> 7.06.2019 02:57 Carlos O'Donell <codonell@redhat.com> wrote:
>>
>> On 6/6/19 5:31 PM, Rafal Luzynski wrote:
>>>>> Possible answers (Cyrillic -> Latin Extended -> ASCII):
>>>>>
>>>>> 1. "Ш" -> "Š" -> "SH"
>>>>>
>>>>> e.g.: "Шема" -> "Šema" -> "SHema"
>>>>> "СÑ
ема" ----------> "Shema"
>>>>>
>>>>> 2. "Ш" -> "Š" -> "Sh"
>>>>>
>>>>> e.g.: "Шема" -> "Šema" -> "Shema"
>>>>> "СÑ
ема" ----------> "Shema"
>>>>>
>>>>> Personally I don't like the answer 1. because "SHema" looks weird
>>>>> to me. Egor in turn does not like the answer 2. because the output
>>>>> string becomes ambiguous.
>>>>>
>>>>> Should we maybe have a smart algorithm which would select the title
>>>>> case or the upper case of the output characters depending on the
>>>>> context in the word? Note that it would not resolve the problem of
>>>>> the output text being ambiguous.
>>>>
>>>> It seems clear that there is no one right/wrong answer but it's a
>>>> matter
>>>> of preference, especially the way this currently works. It might be an
>>>> improvement to output (for instance) SH instead of Sh if all the other
>>>> letters of a word are upper-case as well but not sure what would help
>>>> with the result being unambiguous.
>>>
>>> I think you refer to the idea of implementing a smart algorithm which
>>> would
>>> adapt the lower/upper case depending on the context but indeed it would
>>> not resolve the problem of ambiguity.
>>>
>>> So, the smart algorithm aside, what should be the preferred
>>> transliteration
>>> rule?
>>
>> I have a weak preference for 1. However, I would change my preference if
>> someone showed me existing prior implementations that did 1 or 2.
>
> uconv implements a smart algorithm to adjust the upper/lower case:
>
> ==================================================================
> $ echo "СÑ
ема" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> Skhema
>
> $ echo "Шема" | uconv -f UTF-8 -t ASCII -x Russian-Latin/BGN
> Shema
>
> $ echo "ШÐÐÐ" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> SHEMA
>
> $ echo "ШÐма" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> SHEma
>
> $ echo "Ш Ðма" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN
> SH Yema
> ==================================================================
>
> Also for them it is easier because they decided that "Ð¥" should be
> transliterated to "KH" (I think this is the common thing when
> transliterating to English) while ISO 9 says it should be transliterated
> to "H" and GOST says it should be "X". We can't implement this
> fallback in glibc because the glibc algorithm is very simple.
*Sigh*
I should have known you could find enough examples that contradict
eachother :-)
I'd like to hear what Egor has to say about the data loss aspects.
--
Cheers,
Carlos.
next prev parent reply other threads:[~2019-06-07 12:36 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-05 6:47 Diego (Egor) Kobylkin
2019-06-05 23:51 ` Rafal Luzynski
2019-06-06 9:42 ` Marko Myllynen
2019-06-06 21:31 ` Rafal Luzynski
2019-06-07 0:58 ` Carlos O'Donell
2019-06-07 9:46 ` Diego (Egor) Kobylkin
2019-06-07 11:11 ` Rafal Luzynski
2019-06-07 10:51 ` Rafal Luzynski
2019-06-07 12:36 ` Carlos O'Donell [this message]
2019-06-07 13:00 ` Diego (Egor) Kobylkin
2019-06-07 21:17 ` Carlos O'Donell
2019-06-10 22:44 ` Rafal Luzynski
2019-06-17 8:59 ` Diego (Egor) Kobylkin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=761147fe-75d8-fbbf-b75a-1b58323254f9@redhat.com \
--to=codonell@redhat.com \
--cc=digitalfreak@lingonborough.com \
--cc=egor@kobylkin.com \
--cc=libc-alpha@sourceware.org \
--cc=libc-locales@sourceware.org \
--cc=mfabian@redhat.com \
--cc=myllynen@redhat.com \
--cc=siddhesh@gotplt.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).