[Bug libc/2648] New: localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[Bug libc/2648] New: localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

See also 

http://sourceware.org/bugzilla/show_bug.cgi?id=388

where a very similar bug was reported for Polish. Apparently 
<U0020> (space) is sorted incorrectly in many locales currently.

-- 
           Summary: localedata/locales/es_ES has incorrect LC_COLLATE
                    <space> handling
           Product: glibc
           Version: 2.4
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper at redhat dot com
        ReportedBy: mfabian at suse dot de
                CC: glibc-bugs at sources dot redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |barbier at linuxfr dot org


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2006-05-09 15:53 -------
The change done in http://sourceware.org/bugzilla/show_bug.cgi?id=388
for Polish would be good for Spanish as well.

And not only for Spanish but probably for most, if not all locales.
 

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2006-05-09 15:52 -------
See also:

http://sources.redhat.com/bugzilla/show_bug.cgi?id=374

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2006-05-09 15:55 -------
There is some information concerning this problem in

https://bugzilla.novell.com/show_bug.cgi?id=152778

Unfortunately this bug is not publicly readable.

Therefore I am going to paste the relevant information from 
novell bugzilla here.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2006-05-09 16:00 -------
Original comment in the Novell bugzilla:  When LC_COLLATE=es_ES, the sort command ignores spaces in its sorting algorithm, so it sorts         MAS PUJADAS, FRANCESC after         MASOLIVER GARCIA, JAIME  instead of before, even though the comments in /usr/share/i18n/locales/es_ES indicate that the sorting algorithm for this locales should take spaces into account (and sort them before punctuation characters, numbers and letters).  This spanish customer is not using LC_COLLATE="POSIX" because the sort  command gives incorrect results when dealing with characters with spanish  accents so he has to use LC_COLLATE="es_ES.UTF-8" which is ignoring spaces.  Even /usr/share/i18n/locales/es_ES states:  LC_COLLATE  % Base collation scheme: 1994-03-22  % Ordering algorithm: %  1. Spaces and hyphen (but not soft %     hyphen) before punctuation %     characters, punctuation characters %     before numbers, %     numbers before letters.  I also tested it with every other language setting and the results are always the same:  mortlach:~ # export LC_COLLATE="POSIX" mortlach:~ # sort demo  AB CDESY ABC DETZ ABCD ETX  mortlach:~ # export LC_COLLATE="en_GB.UTF-8" mortlach:~ # sort demo  AB CDESY ABCD ETX ABC DETZ  mortlach:~ # export LC_COLLATE="de_DE.UTF-8" mortlach:~ # sort demo  AB CDESY ABCD ETX ABC DETZ  So the question is why LC_COLLATE="POSIX" behaves differently to any other language setting, if this is a feature where is it documented and why is it so? It doesn't make sence that LC_COLLATE="POSIX" behaves different to the English settings (UK & US) which on the other hand behave exactly the same way as any other language setting so there must be a reason why this is so

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2006-05-09 16:00 -------
Original comment in the Novell bugzilla:

When LC_COLLATE=es_ES, the sort command ignores spaces in its sorting
algorithm, so it sorts
        MAS PUJADAS, FRANCESC
after
        MASOLIVER GARCIA, JAIME 
instead of before, even though the comments in
/usr/share/i18n/locales/es_ES indicate that the sorting algorithm for this
locales should take spaces into account (and sort them before punctuation
characters, numbers and letters).

This spanish customer is not using LC_COLLATE="POSIX" because the sort  command
gives incorrect results when dealing with characters with spanish 
accents so he has to use LC_COLLATE="es_ES.UTF-8" which is ignoring spaces. 
Even /usr/share/i18n/locales/es_ES states:

LC_COLLATE

% Base collation scheme: 1994-03-22

% Ordering algorithm:
%  1. Spaces and hyphen (but not soft
%     hyphen) before punctuation
%     characters, punctuation characters
%     before numbers,
%     numbers before letters.

I also tested it with every other language setting and the results are always
the same:

mortlach:~ # export LC_COLLATE="POSIX"
mortlach:~ # sort demo

AB CDESY
ABC DETZ
ABCD ETX

mortlach:~ # export LC_COLLATE="en_GB.UTF-8"
mortlach:~ # sort demo

AB CDESY
ABCD ETX
ABC DETZ

mortlach:~ # export LC_COLLATE="de_DE.UTF-8"
mortlach:~ # sort demo

AB CDESY
ABCD ETX
ABC DETZ

So the question is why LC_COLLATE="POSIX" behaves differently to any other
language setting, if this is a feature where is it documented and why is it so?
It doesn't make sence that LC_COLLATE="POSIX" behaves different to the English
settings (UK & US) which on the other hand behave exactly the same way as any
other language setting so there must be a reason why this is so


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2006-05-09 16:01 -------
Sorry for the bad formatting in comment #4, apparently
I have a problem with my browser.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]


------- Additional Comments From mfabian at suse dot de  2006-05-09 16:02 -------
Comment from Markus Kuhn from the Novell Bugzilla:

Comment #4 From Markus Kuhn 2006-03-21 11:24 MST
[ ] Private

Glibc implements a 4-pass sorting algorithm, something like the Unicode
Collation Algorithm defined at

  http://www.unicode.org/reports/tr10/

or equivalently the International Standard Ordering defined in ISO 14651. The
SPACE is not ignored, it affects the sorting order only with lower priority
than 

  - the base characters
  - accents
  - whether base characters are uppercase or lower case

At level 4, space is treated like punctuation.

The Unicode sorting algorithm has lots of options. If you look at

  http://www.unicode.org/reports/tr10/#Variable_Weighting

you will see that variable weighting options are avaliable for characters such
as SPACE. Perhaps the UTF-8 locales were configured to use something equivalent
to the "blanked" option, whereas what the user expects here is the
"non-ignorable" option?

It is up to the locale designer to chose these options, and I suspect the
necessary discussion on which options are best here has never taken place.

The culprit is probably in the file

  /usr/share/i18n/locales/iso14651_t1

the line

  <U0020> IGNORE;IGNORE;IGNORE;<U0020> # 32 <SP>

which says that SPACE is sorted at level 4 only, i.e. with lowest priority. I
don't think this is a particularly good choice.

File format spec:
http://www.cl.cam.ac.uk/~mgk25/volatile/ISO-14652.pdf

People like Ulrich Drepper, Alain LaBonté, Keld J. Simonsen would know more on
the origins of this.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2006-05-09 16:04 -------
Another comment by Markus Kuhn:

Comment #11 From Markus Kuhn 2006-04-10 03:49 MST
[ ] Private

Options:

a) Tell the customer that you are sorry that glibc does not at present offer
what he expects, and that we are unable to fix this ourselved without breaking
compatibility with every other glibc-based distribution. One possible
customer-side workaround is to replace SP with NBSP (0xa0) before sorting. NBSP
does already seem to get sorted in the way in which the customer expects SP to
be sorted.

b) Patch in /usr/share/i18n/locales/iso14651_t1 the line

  <U0020> IGNORE;IGNORE;IGNORE;<U0020> # 32 <SP>

to something like

  <U0020> <U0020>;<BAS>;<MIN>;IGNORE

to make SP sort like NBSP does already

WARNING: This obviously breaks compatibility with other Linux distributions.

c) Fund a proper research project aimed as cleaning up the mess that the
collation implementation and configuration is at present in glibc and POSIX,
possibly also investigating user needs and developing a new API for customizing
the sorting order at run-time via environment variables and/or new library
calls.

The reported propblem is only one symptom of the fact that the collating code
(and perhaps even the underlying POSIX spec!) is not really finished and is at
present not properly maintained.

Option c) is perhaps what really should be done, but needs far wider discussion
(beyond Novell) and escalation to management, because someone will have to
spend many weeks (if not months) on sorting this entire issue out properly.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2006-05-10 15:18 -------
It's complete BS to say that spaces are mishandled in most locales.  This was
appropriately researched by the ISO 14651 working group and I trust those people
more than any random user.

It is further completely unacceptable to open one bug and complain about a
million things.

To get anything changed, you have to provide statemsnts from the language
authorities about the proposed change.  If you cannot provide this there
obviously is at least room for discussion and no change is the right approach.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2007-02-17 18:44 -------
No reply in 9 months.  Closing.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |WORKSFORME


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2007-02-23 15:26 -------
By the way, the original bug report in the novell bugzilla is 
publicly readable now:

https://bugzilla.novell.com/show_bug.cgi?id=152778

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[mfabian at suse dot de]

------- Additional Comments From mfabian at suse dot de  2007-02-23 15:53 -------
Why did you fix the similar problem in Polish locale reported in

http://sourceware.org/bugzilla/show_bug.cgi?id=388

?

Was the information supplied in that bug report so much better?

And what about the comment in /usr/share/i18n/locales/es_ES:

LC_COLLATE

% Base collation scheme: 1994-03-22

% Ordering algorithm:
%  1. Spaces and hyphen (but not soft
%     hyphen) before punctuation
%     characters, punctuation characters
%     before numbers,
%     numbers before letters.

This comment doesn't agree with the current implementation, does it?


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WORKSFORME                  |


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[Markus dot Kuhn at cl dot cam dot ac dot uk]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |Markus dot Kuhn at cl dot
                   |                            |cam dot ac dot uk


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[barbier at linuxfr dot org]

------- Additional Comments From barbier at linuxfr dot org  2007-02-27 21:41 -------
> Why did you fix the similar problem in Polish locale reported in
>   http://sourceware.org/bugzilla/show_bug.cgi?id=388
> ?
> 
> Was the information supplied in that bug report so much better?

In BZ388, bug submitter provided references.  In your bug report,
there is none.  One can argue than native speakers know collation
rules for their language, but at least for French this is obviously
wrong ;)

> And what about the comment in /usr/share/i18n/locales/es_ES:
>
> % Ordering algorithm:
> %  1. Spaces and hyphen (but not soft
> %     hyphen) before punctuation
> %     characters, punctuation characters
> %     before numbers,
> %     numbers before letters.
>
> This comment doesn't agree with the current implementation, does it?

Right, AFAICT these collation rules came from an earlier draft of
iso14651_t1, coefficients for U0020 had been fixed but this comment
has not been updated.

Anyway I came to a bookshop today and had a look at a Spanish
dictionary.  Spaces were significant in this dictionary:
  san
  san bernardo
  sana-something, I do not remember the exact word
IMHO you can try to either find references as in BZ388, or give
detailed references (ie. not as I did above, I had no pen to write
down details) to several well known Spanish dictionaries which sort
words as you describe.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[drepper at redhat dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |WAITING


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

[Bug libc/2648] localedata/locales/es_ES has incorrect LC_COLLATE <space> handling

[drepper at redhat dot com]

------- Additional Comments From drepper at redhat dot com  2008-04-08 00:55 -------
No reactions.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |WONTFIX


http://sourceware.org/bugzilla/show_bug.cgi?id=2648

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.