Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Takashi Yano]

Hi,

In the latest cygwin1.dll snapshot, the date field of "ls -l" command
is garbled if LANG=ja_JP.UTF-8.

Other fields seem to be displayed correctly even if Unicode characters
are used. Moreover, other commands which show Unicode characters have
no problem.

This problem causes after the following commit.

commit 41f72ab4d7c404b8ac2a5e1187d79164992e4483 (HEAD)
Author: Thomas Wolff <mintty@users.noreply.github.com>
Date:   Fri Mar 9 13:30:33 2018 +0100

    use generated character data

    The tow* functions use an included case conversion table which can be
    generated from Unicode data.
    The isw* functions use a character categories table (provided by
    categories.c) which can be generated from Unicode data.
    Delegation between current-locale and specific-locale-dependent functions
    was reverted towards the generic locale-dependent functions (*_l.c);
    this is however only relevant on systems with non-Unicode wide character
    locales, thus not on Cygwin.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Takashi Yano]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

On Sat, 23 Jun 2018 02:47:37 +0900
Takashi Yano wrote:
> In the latest cygwin1.dll snapshot, the date field of "ls -l" command
> is garbled if LANG=ja_JP.UTF-8.

I looked into this problem, and found this is caused by incorrect
return value of iswprint().

A simple test case is attached.
The expected result of the test case is: 111111
However, in the latest snapshot, result is: 111000

I think Thomas can easily fix this.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: chk_iswprint.c --]
[-- Type: text/x-csrc; name="chk_iswprint.c", Size: 275 bytes --]

#include <stdio.h>
#include <locale.h>
#include <wchar.h>
#include <wctype.h>

int main()
{
	wchar_t w[] = L"ABC日本語";
	wchar_t *p;

	setlocale(LC_ALL, "ja_JP.UTF-8");
	for (p=w; *p; p++) {
		printf("%d", iswprint(*p));
	}
	printf("\n");
	return 0;
}


[-- Attachment #3: Type: text/plain, Size: 219 bytes --]


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Takashi Yano]

[-- Attachment #1: Type: text/plain, Size: 331 bytes --]

On Sat, 23 Jun 2018 05:39:27 +0900
Takashi Yano wrote:
> A simple test case is attached.
> The expected result of the test case is: 111111
> However, in the latest snapshot, result is: 111000

The test case may be garbled at Japanese string.
I will compress it with gzip and attach it.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

[-- Attachment #2: chk_iswprint.c.gz --]
[-- Type: application/octet-stream, Size: 221 bytes --]

[-- Attachment #3: Type: text/plain, Size: 219 bytes --]


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Takashi Yano]

On Sat, 23 Jun 2018 05:39:27 +0900
Takashi Yano wrote:
> I looked into this problem, and found this is caused by incorrect
> return value of iswprint().

I have found the cause. That is, file categories.t is not correct.

For example,  http://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt
says:

3400;<CJK Ideograph Extension A, First>;Lo;0;L;;;;;N;;;;;
4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;
...
4E00;<CJK Ideograph, First>;Lo;0;L;;;;;N;;;;;
9FEF;<CJK Ideograph, Last>;Lo;0;L;;;;;N;;;;;

However, categories.t is:
    {CAT_Lo, 0x3400, 0},
    {CAT_Lo, 0x4DB5, 0},
...
    {CAT_Lo, 0x4E00, 0},
    {CAT_Lo, 0x9FEA, 0},

Therefore, the script mkcategories which generates categories.t
should be fixed.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Brian Inglis]

On 2018-06-22 17:06, Takashi Yano wrote:
> On Sat, 23 Jun 2018 05:39:27 +0900
> Takashi Yano wrote:
>> I looked into this problem, and found this is caused by incorrect
>> return value of iswprint().
> 
> I have found the cause. That is, file categories.t is not correct.
> 
> For example,  http://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt
> says:
> 
> 3400;<CJK Ideograph Extension A, First>;Lo;0;L;;;;;N;;;;;
> 4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;
> ...
> 4E00;<CJK Ideograph, First>;Lo;0;L;;;;;N;;;;;
> 9FEF;<CJK Ideograph, Last>;Lo;0;L;;;;;N;;;;;
> 
> However, categories.t is:
>     {CAT_Lo, 0x3400, 0},
>     {CAT_Lo, 0x4DB5, 0},
> ...
>     {CAT_Lo, 0x4E00, 0},
>     {CAT_Lo, 0x9FEA, 0},
> 
> Therefore, the script mkcategories which generates categories.t
> should be fixed.

Why are the categories entries not generated from UnicodeData.txt by an (awk)
script?
These entries change with every Unicode release, and a new one came out a few
weeks ago, updated here yesterday.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Thomas Wolff]

Am 23.06.2018 um 20:46 schrieb Brian Inglis:
> On 2018-06-22 17:06, Takashi Yano wrote:
>> On Sat, 23 Jun 2018 05:39:27 +0900
>> Takashi Yano wrote:
>>> I looked into this problem, and found this is caused by incorrect
>>> return value of iswprint().
>> I have found the cause. That is, file categories.t is not correct.
>>
>> For example,  http://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt says:
>>
>> 3400;<CJK Ideograph Extension A, First>;Lo;0;L;;;;;N;;;;;
>> 4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;
>> ...
>> 4E00;<CJK Ideograph, First>;Lo;0;L;;;;;N;;;;;
>> 9FEF;<CJK Ideograph, Last>;Lo;0;L;;;;;N;;;;;
>>
>> However, categories.t is:
>>      {CAT_Lo, 0x3400, 0},
>>      {CAT_Lo, 0x4DB5, 0},
>> ...
>>      {CAT_Lo, 0x4E00, 0},
>>      {CAT_Lo, 0x9FEA, 0},
>>
>> Therefore, the script mkcategories which generates categories.t should be fixed.
Obviously. I will check why the script was failing here and thanks for
the patch already.

> Why are the categories entries not generated from UnicodeData.txt by an (awk) script?
They are generated, with a shell script. They were hard-coded before my
patch.
Why not awk? Because I am not familiar with awk.
> These entries change with every Unicode release, and a new one came out a few
> weeks ago, updated here yesterday.

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Steven Penny]

On Sun, 24 Jun 2018 00:32:59, Thomas Wolff wrote:
> Why not awk? Because I am not familiar with awk.

Should be a simple script, something like this:

$ awk -F ';' '{printf "{CAT_%s, 0x%s, %d},\n", $3, $1, $4}' UnicodeData.txt
{CAT_Lo, 0x3400, 0},
{CAT_Lo, 0x4DB5, 0},
{CAT_Lo, 0x4E00, 0},
{CAT_Lo, 0x9FEF, 0},


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Brian Inglis]

On 2018-06-24 06:27, Steven Penny wrote:
> On Sun, 24 Jun 2018 00:32:59, Thomas Wolff wrote:
>> Why not awk? Because I am not familiar with awk.

Suggestion was awk having suitable capabilities, but perl, python, ruby, or any
other would do.

> Should be a simple script, something like this:
> $ awk -F ';' '{printf "{CAT_%s, 0x%s, %d},\n", $3, $1, $4}' UnicodeData.txt
> {CAT_Lo, 0x3400, 0},
> {CAT_Lo, 0x4DB5, 0},
> {CAT_Lo, 0x4E00, 0},
> {CAT_Lo, 0x9FEF, 0},

A little bit more complex to roll up category ranges, and handle
<... {Fir,La}st> ranges.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Thomas Wolff]

Am 24.06.2018 um 00:32 schrieb Thomas Wolff:
> Am 23.06.2018 um 20:46 schrieb Brian Inglis:
>> On 2018-06-22 17:06, Takashi Yano wrote:
>>> On Sat, 23 Jun 2018 05:39:27 +0900
>>> Takashi Yano wrote:
>>>> I looked into this problem, and found this is caused by incorrect
>>>> return value of iswprint().
>>> I have found the cause. That is, file categories.t is not correct.
>>>
>>> For example, http://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt 
>>> says:
>>>
>>> 3400;<CJK Ideograph Extension A, First>;Lo;0;L;;;;;N;;;;;
>>> 4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;
>>> ...
>>> 4E00;<CJK Ideograph, First>;Lo;0;L;;;;;N;;;;;
>>> 9FEF;<CJK Ideograph, Last>;Lo;0;L;;;;;N;;;;;
>>>
>>> However, categories.t is:
>>>      {CAT_Lo, 0x3400, 0},
>>>      {CAT_Lo, 0x4DB5, 0},
>>> ...
>>>      {CAT_Lo, 0x4E00, 0},
>>>      {CAT_Lo, 0x9FEA, 0},
>>>
>>> Therefore, the script mkcategories which generates categories.t 
>>> should be fixed.
> Obviously. I will check why the script was failing here and thanks for 
> the patch already.
Yeah, I totally failed to consider the First...Last ranges in this 
script, how embarassing.
I would have chosen other markers than 0/1 for the two cases (maybe 
"firstlast" for the new one), and not put it in the middle, but you 
fixed it anyway; except for the Private Use ranges E000..F8FF, 
F0000..FFFFD, 100000..10FFFD, but their category Co isn't used anyway.
I suggest you submit your patch directly to cygwin-patches@cygwin.com. 
One issue, though:
Apparently, you used a Unicode version other than 10.0 as a reference, 
probably 11.0. Maybe the fixing patch should be based on the same 
version, for better clarity and to reduce the diff (fewer changes in 
categories.t and none in caseconv.t).
We can update to 11.0 in addition, but:
Brian wrote:
>> These entries change with every Unicode release, and a new one came 
>> out a few
>> weeks ago, updated here yesterday.
The library updated to 11.0 is probably icu, which isn't used here. The 
libc generation scripts use the original Unicode files which are 
packaged in unicode-ucd which is still at 10.0.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2495 bytes --]

On Jun 24 18:34, Thomas Wolff wrote:
> Am 24.06.2018 um 00:32 schrieb Thomas Wolff:
> > Am 23.06.2018 um 20:46 schrieb Brian Inglis:
> > > On 2018-06-22 17:06, Takashi Yano wrote:
> > > > On Sat, 23 Jun 2018 05:39:27 +0900
> > > > Takashi Yano wrote:
> > > > > I looked into this problem, and found this is caused by incorrect
> > > > > return value of iswprint().
> > > > I have found the cause. That is, file categories.t is not correct.
> > > > 
> > > > For example,
> > > > http://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt says:
> > > > 
> > > > 3400;<CJK Ideograph Extension A, First>;Lo;0;L;;;;;N;;;;;
> > > > 4DB5;<CJK Ideograph Extension A, Last>;Lo;0;L;;;;;N;;;;;
> > > > ...
> > > > 4E00;<CJK Ideograph, First>;Lo;0;L;;;;;N;;;;;
> > > > 9FEF;<CJK Ideograph, Last>;Lo;0;L;;;;;N;;;;;
> > > > 
> > > > However, categories.t is:
> > > >      {CAT_Lo, 0x3400, 0},
> > > >      {CAT_Lo, 0x4DB5, 0},
> > > > ...
> > > >      {CAT_Lo, 0x4E00, 0},
> > > >      {CAT_Lo, 0x9FEA, 0},
> > > > 
> > > > Therefore, the script mkcategories which generates categories.t
> > > > should be fixed.
> > Obviously. I will check why the script was failing here and thanks for
> > the patch already.
> Yeah, I totally failed to consider the First...Last ranges in this script,
> how embarassing.
> I would have chosen other markers than 0/1 for the two cases (maybe
> "firstlast" for the new one), and not put it in the middle, but you fixed it
> anyway; except for the Private Use ranges E000..F8FF, F0000..FFFFD,
> 100000..10FFFD, but their category Co isn't used anyway.
> I suggest you submit your patch directly to cygwin-patches@cygwin.com.

In theory this belongs to newlib AT sourceware DOT org since it only
affects newlib code, and the non-Cygwin newlib guys would probably like
to see these patches as well :)

> One
> issue, though:
> Apparently, you used a Unicode version other than 10.0 as a reference,
> probably 11.0. Maybe the fixing patch should be based on the same version,
> for better clarity and to reduce the diff (fewer changes in categories.t and
> none in caseconv.t).

Takashi, does your patch on cygwin-patches take this into account?

I think Thomas has a good point here, the generated files should
always match a single version.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Takashi Yano]

Hi Thomas,

On Sun, 24 Jun 2018 18:34:06 +0200
Thomas Wolff wrote:
> I would have chosen other markers than 0/1 for the two cases (maybe 
> "firstlast" for the new one), and not put it in the middle, but you 
> fixed it anyway; except for the Private Use ranges E000..F8FF, 
> F0000..FFFFD, 100000..10FFFD, but their category Co isn't used anyway.

Thanks for your comments.

Mark for "Last":
I will use isNormalOrRangeFirst/isRangeLast instead of 0/1 to clarify
the intent.

Position of "Last" mark:
I put the mark 0/1 in the middle, i.e. before the category mark, so that
uniq command can ignore it with -f option which is used in your script.
If I put it after category mark, uniq command put a blank line before
"Last" entry, which will be replaced with undesirable "range" by sed.

Category "Co":
They are already removed in your original script, so I followed it.

> I suggest you submit your patch directly to cygwin-patches@cygwin.com. 

I will submit a new patch to cygwin-patches@cygwin.com.

> One issue, though:
> Apparently, you used a Unicode version other than 10.0 as a reference, 
> probably 11.0. Maybe the fixing patch should be based on the same 
> version, for better clarity and to reduce the diff (fewer changes in 
> categories.t and none in caseconv.t).
> We can update to 11.0 in addition, but:
> Brian wrote:
> >> These entries change with every Unicode release, and a new one came 
> >> out a few
> >> weeks ago, updated here yesterday.
> The library updated to 11.0 is probably icu, which isn't used here. The 
> libc generation scripts use the original Unicode files which are 
> packaged in unicode-ucd which is still at 10.0.

I will remake a patch based on UnicodeData.txt of version 10.0.

Thank you.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Takashi Yano]

Hi Corinna,

On Mon, 25 Jun 2018 11:14:32 +0200
Corinna Vinschen wrote:
> Takashi, does your patch on cygwin-patches take this into account?
> 
> I think Thomas has a good point here, the generated files should
> always match a single version.

I submitted a new patch taking that point into account to cygwin-patches.

Thanks.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Date field of ls -l command is garbled in latest cygwin1.dll snapshot.

[Thomas Wolff]

Am 25.06.2018 um 13:20 schrieb Takashi Yano:
> Hi Corinna,
>
> On Mon, 25 Jun 2018 11:14:32 +0200
> Corinna Vinschen wrote:
>> Takashi, does your patch on cygwin-patches take this into account?
>>
>> I think Thomas has a good point here, the generated files should
>> always match a single version.
> I submitted a new patch taking that point into account to cygwin-patches.
The patch Corinna forwarded to newlib looks good to me. Sorry for
suggesting the wrong mailing list (and for the bug in the first place...)
Thomas

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple