[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
                 CC|                            |ekobylkin at paypal dot com
         Resolution|WORKSFORME                  |---

--- Comment #5 from Egor Kobylkin <ekobylkin at paypal dot com> ---
I would like to try to supply the data you need to make the Cyrillic
transliteration work for the ru_RU locale. Could you point me to an example of
the data you would need need?

Here is what I have tried just to see what works.
$ echo Лаковый |LANG=ru_RU.UTF-8 iconv -t ASCII//TRANSLIT
iconv: (stdin):1:0: cannot convert

$ echo Лаковый |LANG=sr_CS.UTF-8 iconv -t ASCII//TRANSLIT
iconv: (стдул):1:0: не може претворити

$ echo Лаковый |LANG=de_DE.UTF-8 iconv -t ASCII//TRANSLIT
iconv: (Standard-Eingabe):1:0: Kann nicht umwandeln.

$ echo Müßte |LANG=de_DE.UTF-8 iconv -t ASCII//TRANSLIT
M"usste

$ echo Лаковый |LANG=en_US.UTF-8 iconv -t ASCII//TRANSLIT
iconv: (stdin):1:0: cannot convert

$ echo Müßte |LANG=en_US.UTF-8 iconv -t ASCII//TRANSLIT
M"usste

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-29668-listarch-glibc-bugs=sources.redhat.com@sourceware.org Mon Sep 07 11:51:19 2015
Return-Path: <glibc-bugs-return-29668-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 83989 invoked by alias); 7 Sep 2015 11:51:19 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 83927 invoked by uid 48); 7 Sep 2015 11:51:15 -0000
From: "schwab@linux-m68k.org" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug localedata/2872] Transliteration Cyrillic -> ASCII fails
Date: Mon, 07 Sep 2015 11:51:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: localedata
X-Bugzilla-Version: 2.3.6
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: schwab@linux-m68k.org
X-Bugzilla-Status: WAITING
X-Bugzilla-Resolution:
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: libc-locales at sourceware dot org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: security-
X-Bugzilla-Changed-Fields: bug_status
Message-ID: <bug-2872-131-CNDD8L9Y1F@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-2872-131@http.sourceware.org/bugzilla/>
References: <bug-2872-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2015-09/txt/msg00067.txt.bz2
Content-length: 1099

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |WAITING

--- Comment #6 from Andreas Schwab <schwab@linux-m68k.org> ---
Make sure you are not using any local modifications.

$ echo Лаковый |LANG=ru_RU.UTF-8 iconv -t ASCII//TRANSLIT
???????
$ echo Лаковый |LANG=sr_CS.UTF-8 iconv -t ASCII//TRANSLIT
iconv: illegal input sequence at position 0
$ echo Лаковый |LANG=de_DE.UTF-8 iconv -t ASCII//TRANSLIT
???????
$ echo Müßte |LANG=de_DE.UTF-8 iconv -t ASCII//TRANSLIT
Muesste
$ echo Лаковый |LANG=en_US.UTF-8 iconv -t ASCII//TRANSLIT
???????
$ echo Müßte |LANG=en_US.UTF-8 iconv -t ASCII//TRANSLIT
Musste

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-29669-listarch-glibc-bugs=sources.redhat.com@sourceware.org Mon Sep 07 12:16:10 2015
Return-Path: <glibc-bugs-return-29669-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 122278 invoked by alias); 7 Sep 2015 12:16:10 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 122230 invoked by uid 48); 7 Sep 2015 12:16:06 -0000
From: "fweimer at redhat dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug dynamic-link/18928] LD_POINTER_GUARD is not ignored for privileged binaries
Date: Mon, 07 Sep 2015 12:16:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: dynamic-link
X-Bugzilla-Version: unspecified
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: fweimer at redhat dot com
X-Bugzilla-Status: NEW
X-Bugzilla-Resolution:
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: unassigned at sourceware dot org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: security+
X-Bugzilla-Changed-Fields: cc flagtypes.name
Message-ID: <bug-18928-131-wNNfOp7awA@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-18928-131@http.sourceware.org/bugzilla/>
References: <bug-18928-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2015-09/txt/msg00068.txt.bz2
Content-length: 453

https://sourceware.org/bugzilla/show_bug.cgi?id\x18928

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com
              Flags|                            |security+

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #7 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Andreas, in your example the Cyrillic transliteration does not work either.  My
understanding is that the tool is lacking a translation table for Cyrillic to
TRANSLIT for example in the ru_RU locale. This is what Ulrich Drepper asks for
in his comment2 here: https://sourceware.org/bugzilla/show_bug.cgi?id=2872#c2
I would like to know in which form this data should be provided?

I am only concerned with the Cyrillic for now. German serves as an example that
the functionality works at all in at least one case.

My first submission is from Cygwin on Windows 7. While it may be indeed some
effect of the Cygwin (name similarity to Cyrillic is coincidental) I have just
tried the same on Ubuntu 12 for essentially same effect.

$ echo Лаковый |LANG=ru_RU.UTF-8 iconv -t ASCII//TRANSLIT
iconv: illegal input sequence at position 0
$ echo Лаковый |LANG=sr_CS.UTF-8 iconv -t ASCII//TRANSLIT
iconv: illegal input sequence at position 0
$ echo Лаковый |LANG=de_DE.UTF-8 iconv -t ASCII//TRANSLIT
iconv: illegal input sequence at position 0
$ echo Müßte |LANG=de_DE.UTF-8 iconv -t ASCII//TRANSLIT
Miconv: illegal input sequence at position 1
$ echo Лаковый |LANG=en_US.UTF-8 iconv -t ASCII//TRANSLIT
???????
$ echo Müßte |LANG=en_US.UTF-8 iconv -t ASCII//TRANSLIT
Musste

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-29672-listarch-glibc-bugs=sources.redhat.com@sourceware.org Mon Sep 07 15:07:16 2015
Return-Path: <glibc-bugs-return-29672-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 53286 invoked by alias); 7 Sep 2015 15:07:15 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 53226 invoked by uid 48); 7 Sep 2015 15:07:12 -0000
From: "myllynen at redhat dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug localedata/2872] Transliteration Cyrillic -> ASCII fails
Date: Mon, 07 Sep 2015 15:07:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: localedata
X-Bugzilla-Version: 2.3.6
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: myllynen at redhat dot com
X-Bugzilla-Status: WAITING
X-Bugzilla-Resolution:
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: libc-locales at sourceware dot org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: security-
X-Bugzilla-Changed-Fields: cc
Message-ID: <bug-2872-131-xtI1mbtrxw@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-2872-131@http.sourceware.org/bugzilla/>
References: <bug-2872-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2015-09/txt/msg00071.txt.bz2
Content-length: 907

https://sourceware.org/bugzilla/show_bug.cgi?id(72

Marko Myllynen <myllynen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |myllynen at redhat dot com

--- Comment #8 from Marko Myllynen <myllynen at redhat dot com> ---
You'll need to setup testing environment where you can see how your changes
affect to iconv and then try to come up with proper rules, the following links
should help get you started (and are pretty much all the documentation there
is):

https://sourceware.org/glibc/wiki/Locales
http://man7.org/linux/man-pages/man1/iconv.1.html
https://sourceware.org/bugzilla/show_bug.cgi?id\x16061
https://sourceware.org/ml/libc-alpha/2015-07/msg00836.html

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #8585|the OpenOffice Calc         |the LibreOffice Calc
        description|spreadsheet used to create  |spreadsheet used to create
                   |the translit_cyrillic file  |the translit_cyrillic file

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #9 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Created attachment 8585
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8585&action=edit
the OpenOffice Calc spreadsheet used to create the translit_cyrillic file

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #10 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Created attachment 8586
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8586&action=edit
translation table for transliteration of cyrillic to ascii

Single character version. Up to three characters are required to do a
reversible transliteration. The table for a reversible transliteration can be
created through the same spreadsheet included here.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #11 from Egor Kobylkin <ekobylkin at paypal dot com> ---
I have read the linked documents from Marko Myllynen Comment 8. 
My understanding so far is that apart from possibly required code parts that
are not clear yet to me there should be a translation table for the
transliteration.

Based on the 
man page http://man7.org/linux/man-pages/man5/locale.5.html
Russian GOST 7.79-2000 official transliteration table
http://transliteration.ru/gost-7-79-2000/
and the Unicode file http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
I have created a single character transliteration table in the form of a
following list
% CYRILLIC CAPITAL LETTER IO
<U0401> <U0059>
% CYRILLIC CAPITAL LETTER A
<U0410> <U0041>
% CYRILLIC CAPITAL LETTER BE
<U0411> <U0042>
% CYRILLIC CAPITAL LETTER VE
<U0412> <U0056>
etc.
First Unicode value is the Cyrillic letter and the second is a corresponding
ASCII symbol.

The file is attached as translit_cyrillic. 
I wonder if it could be useful already for inclusion into the Latin based
locales files via "include" keyword.

Please let me know what you think. Specifically my understanding is that this
is the list that Ulrich Drepper was requesting.

I would be grateful if somebody familiar with the logic behind the
transliteration file structure could outline the missing parts in case the
above is not sufficient to get bootstrap the cyrillic-ascii transliteration.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[myllynen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Marko Myllynen <myllynen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |maiku.fabian at gmail dot com

--- Comment #12 from Marko Myllynen <myllynen at redhat dot com> ---
I don't read Cyrillic but technically the table looks like what would be
expected. I'm CC'ing Mike Fabian who has done the heavy-lifting for bug 16061 -
Mike how does this look like to you?

If you didn't do so already, please test your changes, the earlier mentioned
wiki page and the following man pages should provide all the needed
information.

http://man7.org/linux/man-pages/man1/locale.1.html
http://man7.org/linux/man-pages/man1/localedef.1.html
http://man7.org/linux/man-pages/man7/locale.7.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #13 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Thank you for the feedback, Marko!
I will do the testing as suggested and will supply the multi-character
transliteration as well. While for my purposes a single-character would do, it
should be more practical to have the multi-character one in place.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 4377 bytes --]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #8586|0                           |1
        is obsolete|                            |

--- Comment #14 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Created attachment 8588
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8588&action=edit
a version that works for localedef

I have tested it with the en_GB locale including into the following section
LC_CTYPE
copy "i18n"

translit_start
include "translit_combining";"translit_cyrillic";""
translit_end
END LC_CTYPE

let's copy en_GB to en_TR for the testing purposes and generate the new locale
en_TR.UTF-8 while being in glibc/localedata/locales
I18NPATH=./ localedef -f UTF-8 -i en_TR en_TR.UTF-8
Now we can test the transliteration 

$echo Съешь ещё этих мягких французских булок, да выпей же чаю |LOCPATH=.
LC_ALL=en_TR.UTF-8 LANG=en_TR.UTF-8 iconv -f UTF-8 -t ASCII//TRANSLIT
S`es` esy etix mygkix francuzskix bulok, da vypej ze cay

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-29689-listarch-glibc-bugs=sources.redhat.com@sourceware.org Tue Sep 08 09:57:19 2015
Return-Path: <glibc-bugs-return-29689-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 39860 invoked by alias); 8 Sep 2015 09:57:19 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 39788 invoked by uid 48); 8 Sep 2015 09:57:14 -0000
From: "ekobylkin at paypal dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug localedata/2872] Transliteration Cyrillic -> ASCII fails
Date: Tue, 08 Sep 2015 09:57:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: localedata
X-Bugzilla-Version: 2.3.6
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: ekobylkin at paypal dot com
X-Bugzilla-Status: WAITING
X-Bugzilla-Resolution:
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: libc-locales at sourceware dot org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: security-
X-Bugzilla-Changed-Fields: attachments.isobsolete attachments.created
Message-ID: <bug-2872-131-QKKX2H0yzK@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-2872-131@http.sourceware.org/bugzilla/>
References: <bug-2872-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2015-09/txt/msg00088.txt.bz2
Content-length: 1038

https://sourceware.org/bugzilla/show_bug.cgi?id(72

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #8588|0                           |1
        is obsolete|                            |

--- Comment #15 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Created attachment 8589
  --> https://sourceware.org/bugzilla/attachment.cgi?id…89&actioníit
multi-character transliteration table cyrillic->ascii with fallback to
single-character

I have not tested the fallback to the sigle-character but have included it in
case somebody needs it. I am not sure on how to test it.
The file should ideally be included in all Latin based locales via include in
this section as follows (example)
LC_CTYPE
copy "i18n"

translit_start
include "translit_combining";"translit_cyrillic";""
translit_end
END LC_CTYPE

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #8585|0                           |1
        is obsolete|                            |

--- Comment #16 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Created attachment 8590
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8590&action=edit
the LibreOffice Calc spreadsheet used to create the translit_cyrillic file with
milti-character transliteration

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #8589|0                           |1
        is obsolete|                            |

--- Comment #17 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Created attachment 8591
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8591&action=edit
multi-character transliteration table cyrillic->ascii with fallback to
single-character

correction: updated the comment in the file to reflect new spreadsheet and
mutli-character feature

I have not tested the fallback to the sigle-character but have included it in
case somebody needs it. I am not sure on how to test it.
The file should ideally be included in all Latin based locales via include in
this section as follows (example)
LC_CTYPE
copy "i18n"

translit_start
include "translit_combining";"translit_cyrillic";""
translit_end
END LC_CTYPE

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #18 from Egor Kobylkin <ekobylkin at paypal dot com> ---
(In reply to Ulrich Drepper from comment #2)
> Transliteration is locale dependend, there is no way around it:
> 
> Russian/Cyrillic:  &#1043;&#1086;&#1088;&#1073;&#1072;&#1095;&#1105;&#1074;
> 
> German transliteration: Gorbaschow
> 
> English transliteration: Gorbatsov or Gorbatsev
> 
> If you want cyrillic transliteration for the locale you use, provide the
> data.
I want to comment on this to clarify my starting point and ask for suggestions
in case somebody decide to take on further development. For now I believe the
issue is solved well however in a most basic way.

>From the Russian speaking person point of view there are various
transliterations possible for Cyrillic depending on the purpose. A good example
of a multiplicity of such transliterations is listed here
http://transliteration.ru/ However having different characters to represent the
Cyrillic letters they have same phonetic meaning for a Russian-speaking person.
So any of them could be used for all the Latin locales. This is what I propose
as a first approximation to solve this issue. My submission above takes this
approach with the GOST 7.79-2000 transliteration chosen as a basis.

For a non-Russian speaking person a yet different transliteration may make
sense to represent their phonetic rules. This is what Ulrich is referring in
his comment above. 

One could take my table as a basis and create separate transliteration tables
to specific locales. The one I have proposed could then still serve as a
ASCII//TRANSLIT target or be replaced by a most proper one.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |suse-linux at ml082 dot pinguin.un
                   |                            |i.cc

--- Comment #19 from Egor Kobylkin <ekobylkin at paypal dot com> ---
*** Bug 12031 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pere at hungry dot com

--- Comment #20 from Egor Kobylkin <ekobylkin at paypal dot com> ---
*** Bug 89 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #21 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Tested including the Greeklish_transliteraion
https://sourceware.org/bugzilla/attachment.cgi?id=6380 from the duplication of
this bug along with the cyrillic tranlation proposed in this bug.

Copied the file translit_greeklish to glibc/localedata/locales. In a copy of
the en_GB locale en_TR2 changed this section 
LC_CTYPE
copy "i18n"
translit_start
include "translit_combining";""
include "translit_cyrillic";""
include "translit_greeklish";""
translit_end
END LC_CTYPE

generated the en_TR2 locale I18NPATH=./ localedef -f UTF-8 -i en_TR2
../../../en_TR/en_TR2.UTF-8

echo CYRILLIC Съешь ещё этих мягких французских булок, да выпей же чаю GREEK
Ελληνικό Ίδρυμα Ευρωπαϊκής και Εξωτερικής |LOCPATH=.../en_TR/
LC_ALL=en_TR2.UTF-8 LANG=en_TR2.UTF-8 iconv -f UTF-8 -t ASCII//TRANSLIT
CYRILLIC S``esh` eshhyo e`tix myagkix franczuzskix bulok, da vy'pej zhe chayu
GREEK Ellhniko Idryma Eyrwpaikhs kai Ekswterikhs

Test successfull.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-29850-listarch-glibc-bugs=sources.redhat.com@sourceware.org Fri Sep 18 13:07:27 2015
Return-Path: <glibc-bugs-return-29850-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 91065 invoked by alias); 18 Sep 2015 13:07:27 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 90993 invoked by uid 48); 18 Sep 2015 13:07:23 -0000
From: "ekobylkin at paypal dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug localedata/2872] Transliteration Cyrillic -> ASCII fails
Date: Fri, 18 Sep 2015 13:07:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: localedata
X-Bugzilla-Version: 2.3.6
X-Bugzilla-Keywords:
X-Bugzilla-Severity: normal
X-Bugzilla-Who: ekobylkin at paypal dot com
X-Bugzilla-Status: NEW
X-Bugzilla-Resolution:
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: libc-locales at sourceware dot org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: security-
X-Bugzilla-Changed-Fields: bug_status
Message-ID: <bug-2872-131-nuJPk0i0od@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-2872-131@http.sourceware.org/bugzilla/>
References: <bug-2872-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2015-09/txt/msg00249.txt.bz2
Content-length: 536

https://sourceware.org/bugzilla/show_bug.cgi?id(72

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW

--- Comment #22 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Reset the bug status to "NEW", to signify it's ready for review by maintainers
of the library

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[myllynen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #23 from Marko Myllynen <myllynen at redhat dot com> ---
Looks like you're making very good progress here. According to
https://sourceware.org/glibc/wiki/Locales the next step would be to check the
Contribution checklist at
https://sourceware.org/glibc/wiki/Contribution%20checklist and post your
patches to libc-alpha + libc-locales for formal review.

However, please be aware that for example Mike's translit update patch has been
pending for a review for many months already [1] so having the patches included
might take a while. But the first step anyway is to post them to the lists.

1) https://sourceware.org/ml/libc-alpha/2015-09/msg00190.html

Thanks.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #24 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Marco,

thanks for reviewing, I will proceed as you propose.

Just in case you know it would be great to have your advice:
In order to get the translit included into the default C.UTF8 locale what is
the venue to discuss that? 

It is a default Cygwin locale and there is no way to generate an own locale in
Cygwin environment AFAIK. But neither could I re-generate the C.UTF8 from the
original POSIX file on my Ubuntu system to test.

I get the the same error messages as listed here. 
http://ask.debian.net/questions/how-to-generate-a-c-utf-8-locale-in-debian-squeeze

So this appears to be a blocker to generate a patch. It seems the POSIX source
file for C.UTF8 is somehow broken for Ubuntu. Do I need to file another bug for
that or is that by design?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #25 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Pangramms in five languages to test the transliteration.

echo CYRILLIC Съешь ещё этих мягких французских булок, да выпей же чаю GREEK
Ελληνικό Ίδρυμα Ευρωπαϊκής και Εξωτερικής GERMAN Zwölf Boxkämpfer jagen Victor
quer über den großen Sylter Deich FRENCH Dès Noël où un zéphyr haï me vêt de
glaçons würmiens je dîne d’exquis rôtis de bœuf au kir à l’aÿ d’âge mûr \&
cætera SPANISH El veloz murciélago hindú comía feliz cardillo y kiwi, la
cigüeña tocaba el saxofón detrás del palenque de paja|LOCPATH=./
LC_ALL=en_TR2.UTF-8 LANG=en_TR2.UTF-8 iconv -f UTF-8 -t ASCII//TRANSLIT

And the result so you can compare. 
CYRILLIC S``esh` eshhyo e`tix myagkix franczuzskix bulok, da vy'pej zhe chayu
GREEK Ellhniko Idryma Eyrwpaikhs kai Ekswterikhs GERMAN Zwolf Boxkampfer jagen
Victor quer uber den grossen Sylter Deich FRENCH Des Noel ou un zephyr hai me
vet de glacons wurmiens je dine d'exquis rotis de boeuf au kir a l'ay d'age mur
& caetera SPANISH El veloz murcielago hindu comia feliz cardillo y kiwi, la
ciguena tocaba el saxofon detras del palenque de paja

-- 
You are receiving this mail because:
You are on the CC list for the bug.
>From glibc-bugs-return-29857-listarch-glibc-bugs=sources.redhat.com@sourceware.org Fri Sep 18 16:15:41 2015
Return-Path: <glibc-bugs-return-29857-listarch-glibc-bugs=sources.redhat.com@sourceware.org>
Delivered-To: listarch-glibc-bugs@sources.redhat.com
Received: (qmail 84760 invoked by alias); 18 Sep 2015 16:15:41 -0000
Mailing-List: contact glibc-bugs-help@sourceware.org; run by ezmlm
Precedence: bulk
List-Id: <glibc-bugs.sourceware.org>
List-Subscribe: <mailto:glibc-bugs-subscribe@sourceware.org>
List-Post: <mailto:glibc-bugs@sourceware.org>
List-Help: <mailto:glibc-bugs-help@sourceware.org>, <http://sourceware.org/lists.html#faqs>
Sender: glibc-bugs-owner@sourceware.org
Delivered-To: mailing list glibc-bugs@sourceware.org
Received: (qmail 84705 invoked by uid 48); 18 Sep 2015 16:15:37 -0000
From: "wdijkstr at arm dot com" <sourceware-bugzilla@sourceware.org>
To: glibc-bugs@sourceware.org
Subject: [Bug math/15367] Let gcc use __builtin_isinf
Date: Fri, 18 Sep 2015 16:15:00 -0000
X-Bugzilla-Reason: CC
X-Bugzilla-Type: changed
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: glibc
X-Bugzilla-Component: math
X-Bugzilla-Version: unspecified
X-Bugzilla-Keywords:
X-Bugzilla-Severity: enhancement
X-Bugzilla-Who: wdijkstr at arm dot com
X-Bugzilla-Status: RESOLVED
X-Bugzilla-Resolution: FIXED
X-Bugzilla-Priority: P2
X-Bugzilla-Assigned-To: unassigned at sourceware dot org
X-Bugzilla-Target-Milestone: 2.23
X-Bugzilla-Flags: security-
X-Bugzilla-Changed-Fields: bug_status cc resolution target_milestone
Message-ID: <bug-15367-131-BJwI3ZGf5a@http.sourceware.org/bugzilla/>
In-Reply-To: <bug-15367-131@http.sourceware.org/bugzilla/>
References: <bug-15367-131@http.sourceware.org/bugzilla/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-Bugzilla-URL: http://sourceware.org/bugzilla/
Auto-Submitted: auto-generated
MIME-Version: 1.0
X-SW-Source: 2015-09/txt/msg00256.txt.bz2
Content-length: 633

https://sourceware.org/bugzilla/show_bug.cgi?id\x15367

Wilco <wdijkstr at arm dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |wdijkstr at arm dot com
         Resolution|---                         |FIXED
   Target Milestone|---                         |2.23

--- Comment #13 from Wilco <wdijkstr at arm dot com> ---
Fixed as per above checkin

--
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[ekobylkin at paypal dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

Egor Kobylkin <ekobylkin at paypal dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|libc-locales at sourceware dot org |ekobylkin at paypal dot com

--- Comment #26 from Egor Kobylkin <ekobylkin at paypal dot com> ---
Created attachment 8618
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8618&action=edit
test file

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[myllynen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=2872

--- Comment #27 from Marko Myllynen <myllynen at redhat dot com> ---
(In reply to Egor Kobylkin from comment #24)
> 
> Just in case you know it would be great to have your advice:
> In order to get the translit included into the default C.UTF8 locale what is
> the venue to discuss that? 

I think it's best to proceed one step at a time - as said it might take a while
to have even the rules included and then C.UTF-8 would need to be implemented
in upstream (see bug 17318).

> It is a default Cygwin locale and there is no way to generate an own locale
> in Cygwin environment AFAIK. But neither could I re-generate the C.UTF8 from
> the original POSIX file on my Ubuntu system to test.
> 
> I get the the same error messages as listed here. 
> http://ask.debian.net/questions/how-to-generate-a-c-utf-8-locale-in-debian-
> squeeze
> 
> So this appears to be a blocker to generate a patch. It seems the POSIX
> source file for C.UTF8 is somehow broken for Ubuntu. Do I need to file
> another bug for that or is that by design?

These all sound like distribution / downstream related issues which should be
handled there, not in glibc upsptream.

Thanks.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/2872] New: Transliteration Cyrillic -> ASCII fails

[edi at gmx dot de]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]

Hello,

I tried to convert some text from Cyrillic (UTF-8) to ASCII, using the
//translit flag. However, it fails badly, all chars are just replaced with ?. It
seems to be independent from my current locale, I can set en_US.UTF-8 or
de_DE.UTF-8 or ru_RU.UTF-8, it still fails.

Transliteration of latin seems to work, though:

echo Müßte асдфасфд | LANG=de_DE.UTF-8 iconv -f UTF-8 -t ASCII//translit
Muesste ????????
echo Müßte асдфасфд | LANG=fr_FR.UTF-8 iconv -f UTF-8 -t ASCII//translit
Musste ????????

I had a "discussion" with a Debian maintainer of glibc, who indicated that the
problem is in the locale which controls the transliterations... but, from my
POV, there should be a default fallback when there is no other transliteration
scheme. And I remember that it has been working with glibc some months or years ago.

-- 
           Summary: Transliteration Cyrillic -> ASCII fails
           Product: glibc
           Version: 2.3.6
            Status: NEW
          Severity: normal
          Priority: P2
         Component: localedata
        AssignedTo: libc-locales at sources dot redhat dot com
        ReportedBy: edi at gmx dot de
                CC: glibc-bugs at sources dot redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=2872

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[dsegan at gmx dot net]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 415 bytes --]


------- Additional Comments From dsegan at gmx dot net  2006-07-20 10:18 -------
Works for me with sr_CS locale:

$ echo 'Müßte Данило' | LANG=sr_CS.UTF-8 iconv -t ASCII//translit
Musste Danilo

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2872

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2007-02-17 19:24 -------
Transliteration is locale dependend, there is no way around it:

Russian/Cyrillic:  Горбачёв

German transliteration: Gorbaschow

English transliteration: Gorbatsov or Gorbatsev

If you want cyrillic transliteration for the locale you use, provide the data.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WORKSFORME


http://sourceware.org/bugzilla/show_bug.cgi?id=2872

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[edi at gmx dot de]

------- Additional Comments From edi at gmx dot de  2007-02-17 22:17 -------
"WORKSFORME" implies that you cannot reproduce the problem, but... does
transliterating Горбачов work with English or not? (see below) If not, how can
it be "resolved"?

Or what are you trying to say with "provide the data"? That there is no data
yet? I have seen it working some years ago. With correct US-style
transliterations. If it is broken now or data has been lost, then TRANSLIT maybe
should be disabled and throw an error immediately. Currently it produces crap
and AFAICS there is no proper documentation explaining why.

The crap it creates is not even consistent within the same language/country
pair, without .UTF-8 suffix it produces more funny non-sense.

echo Горбачов | LANG=de_DE.UTF-8 iconv -t ASCII//TRANSLIT
????????
echo Горбачов | LANG=de_DE.UTF-8 iconv -t ASCII//TRANSLIT
????????
echo Горбачов | LANG=de_DE iconv -t ASCII//TRANSLIT
??? 3/4 N?????N?? 3/4 ??
echo Горбачов | LANG=en_US iconv -t ASCII//TRANSLIT
iconv: illegal input sequence at position 0



-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WORKSFORME                  |


http://sourceware.org/bugzilla/show_bug.cgi?id=2872

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug localedata/2872] Transliteration Cyrillic -> ASCII fails

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2007-02-19 00:51 -------
It all works as designed given the data provided.  If you want change, provide
the data.  Otherwise go away.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |WORKSFORME


http://sourceware.org/bugzilla/show_bug.cgi?id=2872

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.