From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from omta001.cacentral1.a.cloudfilter.net (omta001.cacentral1.a.cloudfilter.net [3.97.99.32]) by sourceware.org (Postfix) with ESMTPS id D11E73858D35 for ; Fri, 24 Mar 2023 22:49:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D11E73858D35 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=Shaw.ca Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=shaw.ca Received: from shw-obgw-4002a.ext.cloudfilter.net ([10.228.9.250]) by cmsmtp with ESMTP id fkp8paUZZuZMSfqDcpRLf7; Fri, 24 Mar 2023 22:49:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=shaw.ca; s=s20180605; t=1679698148; bh=1ZxtwnTSSJZoiu8xKZYBtWVLssjr6D3Rude4LC6WBEA=; h=Date:Reply-To:Subject:To:References:From:In-Reply-To; b=Gkls+9D4yb8WPlLIi+bSwnfW3IkmQu2kbu7qGyFVlaGBq+ok8YpwwurMylqQoT06a 9cZ1SjwXzMbz6t3VwjNQ1gPRrRmHpZbBw1nk8Y2lT/Fr0phA3eImBNKGbNKB5O+Cu/ m5A97sIeX1hHBtd6RC3VSTv1ZjAZI6DRxHSNk4dva+opuMdpvguZn/+ZxR4l3a+g99 2uRoLOCKoZYbSFVdgIJaASmaphCfdDeCE+JTZH5g6mfov3S1/By9Peu4sCcEg/iJbL 1ojggYoYFNf+PKCiW+DcLJ0IgWuJoD2g9mnAupKqJ6y1kw34k3qeQsqmDZ2T2goyLs 7XGNVrLVW8rKg== Received: from [10.0.0.5] ([184.64.102.149]) by cmsmtp with ESMTP id fqDbplilYyAOefqDbpx6DE; Fri, 24 Mar 2023 22:49:08 +0000 X-Authority-Analysis: v=2.4 cv=e5oV9Il/ c=1 sm=1 tr=0 ts=641e28e4 a=DxHlV3/gbUaP7LOF0QAmaA==:117 a=DxHlV3/gbUaP7LOF0QAmaA==:17 a=IkcTkHD0fZMA:10 a=BqEg4_3jAAAA:8 a=qv0N3cA8AAAA:8 a=n8i27M1mAAAA:8 a=1XWaLZrsAAAA:8 a=Og4VvuUy-h3zDRD9t38A:9 a=QEXdDO2ut3YA:10 a=0mFWnFbQd5xWBqmg7tTt:22 a=IkJeoXH5QaFgIRku97pq:22 Message-ID: Date: Fri, 24 Mar 2023 16:49:07 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Reply-To: cygwin@cygwin.com Subject: Re: newlocale: Linux incompatibility Content-Language: en-CA To: cygwin@cygwin.com References: From: Brian Inglis Organization: Inglis In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-CMAE-Envelope: MS4xfFrfYEuMgDvs2zCd5AJPfCOyU0ntLVe+/yTCsH/zsfSjO9NV54bPd+6F3F38byYQy4UTCP86DCr3GpXRQ24Z1Av+RQY9f8L6Xyj68IvxP7rEAOKygcIC RL/zL9S8swvhiYe14Fd5JLGIrBdDlyE7dlBmuxKl+vuxWqkzDjcH1E9fYLucnH6723zy0FOW5ySVxQ== X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,KAM_SHORT,NICE_REPLY_A,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On 2023-03-24 06:18, Corinna Vinschen via Cygwin wrote: > On Mar 23 22:14, Corinna Vinschen via Cygwin wrote: >> On Mar 23 15:48, Ken Brown via Cygwin wrote: >>> I'm reporting this here rather than the newlib list because the behavior is >>> compatible with Posix but not Linux, so I think it's a Cygwin issue. >> >> Actually, it's a Windows issue :) >> >>> Consider the following test case: >>> >>> $ cat locale_test.c >>> #include >>> #include >>> >>> int main () >>> { >>> const char *locale = "en_DE.UTF-8"; >>> locale_t loc = newlocale (LC_COLLATE_MASK | LC_CTYPE_MASK, locale, 0); >>> if (!loc) >>> perror ("newlocale"); >>> else >>> printf ("newlocale succeeded on invalid locale %s\n", locale); >>> } >>> >>> $ gcc -o locale_test locale_test.c >>> >>> $ ./locale_test.exe >>> newlocale succeeded on invalid locale en_DE.UTF-8 >>> >>> On Linux, the newlocale call fails with ENOENT, as is documented on the man >>> page. Posix doesn't say what should happen on an invalid locale, so this is >>> not, strictly speaking, a bug. >> >> Three bugs in fact. >> >> First, it's a bug in the Emacs testsuite. The test simply assumes that >> there's no en_DE locale on any system, but that's just not true. >> Windows support the RFC 5646 locale "en-DE", which is called "English >> (Germany)" in the "Region" settings. >> >> You can also check with `locale -av | less' and search for en_DE. >> >> For the reminder of this mail, I assume you're talking about Cygwin 3.5. >> I won't fix this for 3.4 anymore, given how much locale handling has >> changed for 3.5. >> >> The second bug is that Cygwin blindly trusts the Windows function >> ResolveLocaleName(). That function blatantly converts even vaguely >> similar locales into something it supports. E.g., it converts "en-XY" >> to "en-US". I. .e., even if you use "en_XY.utf8" as locale, the above >> testcase will wrongly succeed. So I have to rethink how I resolve POSIX >> locales to Windows locales. Does Windows even consider https://www.rfc-editor.org/rfc/rfc4647 "Matching of Language Tags", part of https://www.rfc-editor.org/info/bcp47 "Language Tags", and if POSIX only matches exactly, will LANGUAGE be able to be used for fallback? I currently define LANGUAGE=en_CA:en_GB:en in case en-CA is unsupported by anything. [I use my own en-CA locale not the glibc default created by https://rap.dk/.] Will "-" be supported like "_" as a separator in values? >> And the third bug is that Cygwin fails to set errno if it doesn't >> support a locale, but that's a minor inconvenience in comparison. >> >> Thanks for the report, I totally missed the above problem with >> ResolveLocaleName. > > I pushed a couple of patches which hopefully clean up the code. It's > really frustrating how these Windows locale functions work. Or, rather, > not work. I mean, come on... > > - ResolveLocaleName() resolves "ff-BF" to "ff-Latn-SN", not to > "ff-Adlm-BF" or "ff-Latn-BF", even though both exist. > > - There's a locale called "sd-Arab-PK" and a locale "sd-Deva-IN". If > you ask for the script used in "sd-IN", the result is "Arab", not > "Deva". > > I had to create a replacement function for ResolveLocaleName which > doesn't return totally screwy and unexpected results, and special case > two more locales in /proc/locales output so the output makes sense. Aha - a nice new 3.5.0 feature - as well as /proc/codesets - is that charsets e.g. ISO-10646, etc. rather than encodings e.g. UTF-8, etc.! FYI Google fixed their English L14N falling back to en-GB except US territories: https://developer.android.com/guide/topics/resources/multilingual-support#postN https://issuetracker.google.com/issues/64429534#comment6 and there have been similar issues posted for other languages. > Oh, and I added error handling to the code so newlocale is now able to > set errno to ENOENT if the locale is not supported. > > If you want to test this, the changes are in test release > 3.5.0-0.260.gb5b67a65f87c, which is just building. -- Take care. Thanks, Brian Inglis Calgary, Alberta, Canada La perfection est atteinte Perfection is achieved non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut -- Antoine de Saint-Exupéry