[Bug localedata/18943] New: Collation of NFD strings

[Bug localedata/18943] New: Collation of NFD strings

[egmont at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18943

            Bug ID: 18943
           Summary: Collation of NFD strings
           Product: glibc
           Version: 2.22
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: localedata
          Assignee: unassigned at sourceware dot org
          Reporter: egmont at gmail dot com
                CC: libc-locales at sourceware dot org
  Target Milestone: ---

Forking off from bug 18927 comment 8 & 11:

Collate definitions currently assume the input to be in NFC. If the available
UTF-8 unittests are converted to NFD (the localedata/*.in files which have
UTF-8 in Makefile's test-input) then they fail.

It would be nice to automatically make normalization the lowest priority factor
when deciding on collation, so that different normalizations of the same word
are as close to each other as possible. That is, to implement it once (e.g. in
iso14651_common) without having to modify individual locale definitions.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [Bug localedata/18943] New: Collation of NFD strings

[Keld Simonsen]

On Wed, Sep 09, 2015 at 07:46:02PM +0000, egmont at gmail dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=18943
> 
>             Bug ID: 18943
>            Summary: Collation of NFD strings
>            Product: glibc
>            Version: 2.22
>             Status: NEW
>           Severity: enhancement
>           Priority: P2
>          Component: localedata
>           Assignee: unassigned at sourceware dot org
>           Reporter: egmont at gmail dot com
>                 CC: libc-locales at sourceware dot org
>   Target Milestone: ---
> 
> Forking off from bug 18927 comment 8 & 11:
> 
> Collate definitions currently assume the input to be in NFC. If the available
> UTF-8 unittests are converted to NFD (the localedata/*.in files which have
> UTF-8 in Makefile's test-input) then they fail.
> 
> It would be nice to automatically make normalization the lowest priority factor
> when deciding on collation, so that different normalizations of the same word
> are as close to each other as possible. That is, to implement it once (e.g. in
> iso14651_common) without having to modify individual locale definitions.

Both NFC and NFD data should collate as expected. And you can mix then as you like,
you do not need to normalize them.

Best regards
keld

[Bug localedata/18943] Collation of NFD strings

[keld at keldix dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18943

--- Comment #1 from keld at keldix dot com <keld at keldix dot com> ---
On Wed, Sep 09, 2015 at 07:46:02PM +0000, egmont at gmail dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=18943
> 
>             Bug ID: 18943
>            Summary: Collation of NFD strings
>            Product: glibc
>            Version: 2.22
>             Status: NEW
>           Severity: enhancement
>           Priority: P2
>          Component: localedata
>           Assignee: unassigned at sourceware dot org
>           Reporter: egmont at gmail dot com
>                 CC: libc-locales at sourceware dot org
>   Target Milestone: ---
> 
> Forking off from bug 18927 comment 8 & 11:
> 
> Collate definitions currently assume the input to be in NFC. If the available
> UTF-8 unittests are converted to NFD (the localedata/*.in files which have
> UTF-8 in Makefile's test-input) then they fail.
> 
> It would be nice to automatically make normalization the lowest priority factor
> when deciding on collation, so that different normalizations of the same word
> are as close to each other as possible. That is, to implement it once (e.g. in
> iso14651_common) without having to modify individual locale definitions.

Both NFC and NFD data should collate as expected. And you can mix then as you
like,
you do not need to normalize them.

Best regards
keld

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18943] Collation of NFD strings

[egmont at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18943

--- Comment #2 from Egmont Koblinger <egmont at gmail dot com> ---
(In reply to keld@keldix.com from comment #1)

> Both NFC and NFD data should collate as expected. And you can mix then as
> you like,
> you do not need to normalize them.

Not sure what you mean by "should" or "can"... whether you agree with me that
this should be the desired behavior (glad to hear it), or claim that this is
what actually happens (which is unfortunately false).

Revert a broken change pointed out in bug 18589 (to make the tests pass
deterministicly at the first place). Run "make tests" -> success.

Then use "uconv -x any-nfd" to convert fr_FR.in, si_LK.in, tr_TR.in, uk_UA.in
(and perhaps hu_HU.in from bug 18934) to NFD. Re-run "make tests" -> failure.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [Bug localedata/18943] Collation of NFD strings

[Keld Simonsen]

On Thu, Sep 10, 2015 at 06:38:43AM +0000, egmont at gmail dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=18943
> 
> --- Comment #2 from Egmont Koblinger <egmont at gmail dot com> ---
> (In reply to keld@keldix.com from comment #1)
> 
> > Both NFC and NFD data should collate as expected. And you can mix then as
> > you like,
> > you do not need to normalize them.
> 
> Not sure what you mean by "should" or "can"... whether you agree with me that
> this should be the desired behavior (glad to hear it), or claim that this is
> what actually happens (which is unfortunately false).
> 
> Revert a broken change pointed out in bug 18589 (to make the tests pass
> deterministicly at the first place). Run "make tests" -> success.
> 
> Then use "uconv -x any-nfd" to convert fr_FR.in, si_LK.in, tr_TR.in, uk_UA.in
> (and perhaps hu_HU.in from bug 18934) to NFD. Re-run "make tests" -> failure.

What I mean is that ISO 14651 tables are made to have this feature - I specified it.
Whether this is then the case for the i18n locale in glibc, I understand
that this is not the case at this point. Whether this is because of old data
or insufficient implementation of the standard I don't know.

best regards
keld

[Bug localedata/18943] Collation of NFD strings

[keld at keldix dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18943

--- Comment #3 from keld at keldix dot com <keld at keldix dot com> ---
On Thu, Sep 10, 2015 at 06:38:43AM +0000, egmont at gmail dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=18943
> 
> --- Comment #2 from Egmont Koblinger <egmont at gmail dot com> ---
> (In reply to keld@keldix.com from comment #1)
> 
> > Both NFC and NFD data should collate as expected. And you can mix then as
> > you like,
> > you do not need to normalize them.
> 
> Not sure what you mean by "should" or "can"... whether you agree with me that
> this should be the desired behavior (glad to hear it), or claim that this is
> what actually happens (which is unfortunately false).
> 
> Revert a broken change pointed out in bug 18589 (to make the tests pass
> deterministicly at the first place). Run "make tests" -> success.
> 
> Then use "uconv -x any-nfd" to convert fr_FR.in, si_LK.in, tr_TR.in, uk_UA.in
> (and perhaps hu_HU.in from bug 18934) to NFD. Re-run "make tests" -> failure.

What I mean is that ISO 14651 tables are made to have this feature - I
specified it.
Whether this is then the case for the i18n locale in glibc, I understand
that this is not the case at this point. Whether this is because of old data
or insufficient implementation of the standard I don't know.

best regards
keld

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18943] Collation of NFD strings

[egmont at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18943

--- Comment #4 from Egmont Koblinger <egmont at gmail dot com> ---
Thanks for the clarification!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/18943] Collation of NFD strings

[maiku.fabian at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18943

Mike FABIAN <maiku.fabian at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |maiku.fabian at gmail dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.