From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 54313 invoked by alias); 7 Jun 2019 12:36:02 -0000 Mailing-List: contact libc-locales-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: libc-locales-owner@sourceware.org Received: (qmail 54021 invoked by uid 89); 7 Jun 2019 12:36:02 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: =?ISO-8859-1?Q?No, score=1.9 required=5.0 tests=AWL,BAYES_00,BODY_8BITS,GARBLED_BODY,KAM_ASCII_DIVIDERS,KAM_MANYTO,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 spammy==d0=bc=d0=b0, HTo:U*siddhesh, russian-latin, RussianLatin?= X-HELO: mail-qk1-f171.google.com Return-Path: Subject: Re: [PING^8][PATCH v12] Locales: Cyrillic -> ASCII transliteration [BZ #2872] To: Rafal Luzynski , Marko Myllynen , "Diego (Egor) Kobylkin" , "libc-alpha@sourceware.org" , "libc-locales@sourceware.org" , Siddhesh Poyarekar Cc: Mike Fabian References: <2030695416.914859.1559778544120@poczta.nazwa.pl> <1640311749.1550210.1559856673283@poczta.nazwa.pl> <054f3b06-3ca8-00b0-ee07-1ff86a4106dc@redhat.com> <956159024.1658672.1559904734686@poczta.nazwa.pl> From: Carlos O'Donell Message-ID: <761147fe-75d8-fbbf-b75a-1b58323254f9@redhat.com> Date: Fri, 07 Jun 2019 12:36:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.0 MIME-Version: 1.0 In-Reply-To: <956159024.1658672.1559904734686@poczta.nazwa.pl> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-SW-Source: 2019-q2/txt/msg00085.txt.bz2 On 6/7/19 6:52 AM, Rafal Luzynski wrote: > 7.06.2019 02:57 Carlos O'Donell wrote: >> >> On 6/6/19 5:31 PM, Rafal Luzynski wrote: >>>>> Possible answers (Cyrillic -> Latin Extended -> ASCII): >>>>> >>>>> 1. "Ш" -> "Š" -> "SH" >>>>> >>>>> e.g.: "Шема" -> "Šema" -> "SHema" >>>>> "Схема" ----------> "Shema" >>>>> >>>>> 2. "Ш" -> "Š" -> "Sh" >>>>> >>>>> e.g.: "Шема" -> "Šema" -> "Shema" >>>>> "Схема" ----------> "Shema" >>>>> >>>>> Personally I don't like the answer 1. because "SHema" looks weird >>>>> to me. Egor in turn does not like the answer 2. because the output >>>>> string becomes ambiguous. >>>>> >>>>> Should we maybe have a smart algorithm which would select the title >>>>> case or the upper case of the output characters depending on the >>>>> context in the word? Note that it would not resolve the problem of >>>>> the output text being ambiguous. >>>> >>>> It seems clear that there is no one right/wrong answer but it's a >>>> matter >>>> of preference, especially the way this currently works. It might be an >>>> improvement to output (for instance) SH instead of Sh if all the other >>>> letters of a word are upper-case as well but not sure what would help >>>> with the result being unambiguous. >>> >>> I think you refer to the idea of implementing a smart algorithm which >>> would >>> adapt the lower/upper case depending on the context but indeed it would >>> not resolve the problem of ambiguity. >>> >>> So, the smart algorithm aside, what should be the preferred >>> transliteration >>> rule? >> >> I have a weak preference for 1. However, I would change my preference if >> someone showed me existing prior implementations that did 1 or 2. > > uconv implements a smart algorithm to adjust the upper/lower case: > > ================================================================== > $ echo "Схема" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN > Skhema > > $ echo "Шема" | uconv -f UTF-8 -t ASCII -x Russian-Latin/BGN > Shema > > $ echo "ШЕМА" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN > SHEMA > > $ echo "ШЕма" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN > SHEma > > $ echo "Ш Ема" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/BGN > SH Yema > ================================================================== > > Also for them it is easier because they decided that "Х" should be > transliterated to "KH" (I think this is the common thing when > transliterating to English) while ISO 9 says it should be transliterated > to "H" and GOST says it should be "X". We can't implement this > fallback in glibc because the glibc algorithm is very simple. *Sigh* I should have known you could find enough examples that contradict eachother :-) I'd like to hear what Egor has to say about the data loss aspects. -- Cheers, Carlos.