From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 122544 invoked by alias); 29 Oct 2015 22:21:26 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 122534 invoked by uid 89); 29 Oct 2015 22:21:25 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.6 required=5.0 tests=AWL,BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS,SPF_PASS autolearn=ham version=3.3.2 X-HELO: limerock03.mail.cornell.edu Received: from limerock03.mail.cornell.edu (HELO limerock03.mail.cornell.edu) (128.84.13.243) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 29 Oct 2015 22:21:24 +0000 X-CornellRouted: This message has been Routed already. Received: from authusersmtp.mail.cornell.edu (granite3.serverfarm.cornell.edu [10.16.197.8]) by limerock03.mail.cornell.edu (8.14.4/8.14.4_cu) with ESMTP id t9TMLMS2003609 for ; Thu, 29 Oct 2015 18:21:22 -0400 Received: from [10.13.22.3] (50-192-21-217-static.hfc.comcastbusiness.net [50.192.21.217]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id t9TMLLvY029569 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Thu, 29 Oct 2015 18:21:22 -0400 Subject: Re: Bug in collation functions? To: cygwin@cygwin.com References: <563148AF.1000502@cornell.edu> <5631996D.7040908@redhat.com> <20151029075050.GE5319@calimero.vinschen.de> <20151029083057.GH5319@calimero.vinschen.de> <56321815.7000203@cornell.edu> <20151029153516.GJ5319@calimero.vinschen.de> <56323F2E.4030807@cornell.edu> <56324598.9060604@cornell.edu> <56324E82.7000402@redhat.com> <563268A4.6000005@cornell.edu> <56329462.2090206@cornell.edu> From: Ken Brown Message-ID: <56329BE8.808@cornell.edu> Date: Fri, 30 Oct 2015 14:07:00 -0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <56329462.2090206@cornell.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-IsSubscribed: yes X-SW-Source: 2015-10/txt/msg00560.txt.bz2 On 10/29/2015 5:49 PM, Ken Brown wrote: > On 10/29/2015 2:42 PM, Ken Brown wrote: >> On 10/29/2015 12:51 PM, Eric Blake wrote: >>> On 10/29/2015 10:13 AM, Ken Brown wrote: >>> >>>> Never mind. My test case was flawed, because it didn't check for the >>>> possibility that wcscoll might return 0. Here's a revised >>>> definition of >>>> the "compare" function: >>>> >>>> void >>>> compare (const wchar_t *a, const wchar_t *b, const char *loc) >>>> { >>>> setlocale (LC_COLLATE, loc); >>>> int res = wcscoll (a, b); >>>> char c = res < 0 ? '<' : res > 0 ? '>' : '='; >>>> printf ("\"%ls\" %c \"%ls\" in %s locale\n", a, c, b, loc); >>>> } >>>> >>>> With this change (and the use of NORM_IGNORESYMBOLS) the test returns >>>> the following on Cygwin: >>>> >>>> $ ./wcscoll_test >>>> "11" > "1.1" in POSIX locale >>>> "11" = "1.1" in en_US.UTF-8 locale >>>> "11" > "1 2" in POSIX locale >>>> "11" < "1 2" in en_US.UTF-8 locale >>>> >>>> It still differs from Linux, but it's good enough to make the emacs >>>> test >>>> pass. Moreover, this behavior actually seems more reasonable to me >>>> than >>>> the Linux behavior. After all, if you're ignoring punctuation, how can >>>> you decide which of "11" or "1.1" comes first? >>> >>> Careful. POSIX is proposing some wording that say that normal locales >>> should always implement a fallback of last resort (and that locales that >>> do not do so should have a special name including '@', to make it >>> obvious). It is not standardized yet, but worth thinking about. >>> >>> http://austingroupbugs.net/view.php?id=938 >>> http://austingroupbugs.net/view.php?id=963 >>> >>> The intent of that wording is that if ignoring punctuation could cause >>> two strings to otherwise compare equal, the fallback of a total ordering >>> on all characters means that the final result of strcoll() will not be 0 >>> unless the two strings are identical. >> >> In that case, I think Cygwin should start by using NORM_IGNORESYMBOLS in >> non-POSIX locales, with the goal of eventually moving toward emulating >> glibc. I don't know what fallback glibc uses or how hard it would be to >> implement this on Cygwin. > > I withdraw this suggestion. I took a look at the glibc code, and I > don't see any reasonable way for Cygwin to emulate it precisely. On the > other hand, I have an idea for a simple fallback. I'll play with it a > little and then submit a patch. The fallback I had in mind is to return the shorter string if they have different lengths and otherwise to revert to wcscmp. Using this, both Cygwin and Linux give the following comparisons: "11" > "1.1" in POSIX locale "11" < "1.1" in en_US.UTF-8 locale "11" > "1 2" in POSIX locale "11" < "1.2" in en_US.UTF-8 locale "1 1" < "1.1" in POSIX locale "1 1" < "1.1" in en_US.UTF-8 locale If this seems reasonable, I'll test it more extensively and then submit a patch. Ken P.S. In case others want to test this in different locales, here's the patch so far, just for wcscoll: diff --git a/winsup/cygwin/nlsfuncs.cc b/winsup/cygwin/nlsfuncs.cc index f7031f9..c33aa24 100644 --- a/winsup/cygwin/nlsfuncs.cc +++ b/winsup/cygwin/nlsfuncs.cc @@ -1156,10 +1156,15 @@ wcscoll (const wchar_t *__restrict ws1, const wchar_t *__restrict ws2) if (!collate_lcid) return wcscmp (ws1, ws2); - ret = CompareStringW (collate_lcid, 0, ws1, -1, ws2, -1); + ret = CompareStringW (collate_lcid, NORM_IGNORESYMBOLS, ws1, -1, ws2, -1); if (!ret) set_errno (EINVAL); - return ret - CSTR_EQUAL; + ret -= CSTR_EQUAL; + if (!ret) + ret = wcslen (ws1) - wcslen (ws2); + if (!ret) + ret = wcscmp (ws1, ws2); + return ret; } extern "C" int -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple