From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 28210 invoked by alias); 7 Jun 2019 10:51:44 -0000 Mailing-List: contact libc-locales-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: libc-locales-owner@sourceware.org Received: (qmail 28191 invoked by uid 89); 7 Jun 2019 10:51:44 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: =?ISO-8859-1?Q?No, score=-4.1 required=5.0 tests=AWL,BAYES_00,BODY_8BITS,GARBLED_BODY,KAM_ASCII_DIVIDERS,KAM_MANYTO,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 spammy==d0=bc=d0=b0, HTo:U*siddhesh?= X-HELO: shared-ano163.rev.nazwa.pl X-Spam-Score: -1 Date: Fri, 07 Jun 2019 10:51:00 -0000 From: Rafal Luzynski To: Carlos O'Donell , Marko Myllynen , "Diego (Egor) Kobylkin" , "libc-alpha@sourceware.org" , "libc-locales@sourceware.org" , Siddhesh Poyarekar Cc: Mike Fabian Message-ID: <956159024.1658672.1559904734686@poczta.nazwa.pl> In-Reply-To: <054f3b06-3ca8-00b0-ee07-1ff86a4106dc@redhat.com> References: <2030695416.914859.1559778544120@poczta.nazwa.pl> <1640311749.1550210.1559856673283@poczta.nazwa.pl> <054f3b06-3ca8-00b0-ee07-1ff86a4106dc@redhat.com> Subject: Re: [PING^8][PATCH v12] Locales: Cyrillic -> ASCII transliteration [BZ #2872] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SW-Source: 2019-q2/txt/msg00083.txt.bz2 7.06.2019 02:57 Carlos O'Donell wrote: >=20 > On 6/6/19 5:31 PM, Rafal Luzynski wrote: > >>> Possible answers (Cyrillic -> Latin Extended -> ASCII): > >>> > >>> 1. "=D0=A8" -> "=C5=A0" -> "SH" > >>> > >>> e.g.: "=D0=A8=D0=B5=D0=BC=D0=B0" -> "=C5=A0ema" -> "SHema" > >>> "=D0=A1=D1=85=D0=B5=D0=BC=D0=B0" ----------> "Shema" > >>> > >>> 2. "=D0=A8" -> "=C5=A0" -> "Sh" > >>> > >>> e.g.: "=D0=A8=D0=B5=D0=BC=D0=B0" -> "=C5=A0ema" -> "Shema" > >>> "=D0=A1=D1=85=D0=B5=D0=BC=D0=B0" ----------> "Shema" > >>> > >>> Personally I don't like the answer 1. because "SHema" looks weird > >>> to me. Egor in turn does not like the answer 2. because the output > >>> string becomes ambiguous. > >>> > >>> Should we maybe have a smart algorithm which would select the title > >>> case or the upper case of the output characters depending on the > >>> context in the word? Note that it would not resolve the problem of > >>> the output text being ambiguous. > >> > >> It seems clear that there is no one right/wrong answer but it's a > >> matter > >> of preference, especially the way this currently works. It might be an > >> improvement to output (for instance) SH instead of Sh if all the other > >> letters of a word are upper-case as well but not sure what would help > >> with the result being unambiguous. > >=20 > > I think you refer to the idea of implementing a smart algorithm which > > would > > adapt the lower/upper case depending on the context but indeed it would > > not resolve the problem of ambiguity. > >=20 > > So, the smart algorithm aside, what should be the preferred > > transliteration > > rule? >=20 > I have a weak preference for 1. However, I would change my preference if > someone showed me existing prior implementations that did 1 or 2. uconv implements a smart algorithm to adjust the upper/lower case: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D $ echo "=D0=A1=D1=85=D0=B5=D0=BC=D0=B0" | uconv -f UTF-8 -t ASCII -x ru-ru_= Latn/BGN Skhema $ echo "=D0=A8=D0=B5=D0=BC=D0=B0" | uconv -f UTF-8 -t ASCII -x Russian-Lati= n/BGN Shema $ echo "=D0=A8=D0=95=D0=9C=D0=90" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/B= GN SHEMA $ echo "=D0=A8=D0=95=D0=BC=D0=B0" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/B= GN SHEma $ echo "=D0=A8 =D0=95=D0=BC=D0=B0" | uconv -f UTF-8 -t ASCII -x ru-ru_Latn/= BGN SH Yema =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Also for them it is easier because they decided that "=D0=A5" should be transliterated to "KH" (I think this is the common thing when transliterating to English) while ISO 9 says it should be transliterated to "H" and GOST says it should be "X". We can't implement this fallback in glibc because the glibc algorithm is very simple. Regards, Rafal