Cygwin 2.6.0: unreadable UTF-8 in Windows console

Cygwin 2.6.0: unreadable UTF-8 in Windows console

[Ivan Vanyushkin]

Something has changed in version 2.6.0, and now UTF-8 text can't be displayed in Windows console (cmd).

1. Create a file "test.txt" with non-ASCII text in UTF-8 encoding.
2. Run "cmd".
3. Run:

C:\Cygwin\bin\cat test.txt
â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’ 8000 â–’â–’. â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’.

Non-ASCII text is not readable. Older Cygwin 2.5.2 has no such issue.

C:\Cygwin\bin\uname -a
CYGWIN_NT-10.0 PCName 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin

C:\Cygwin\bin\locale
LANG=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

Same issue with any other commands like "grep", or with utilities built and run under Cygwin 2.6.0.
Same issue in other Windows consoles, like ConEmu or FAR Manager.
If I change Windows console encoding to UTF-8 (run: "chcp 65001"), file can be correctly displayed natively
(run: "type test.txt"), but Cygwin "cat" still has the same issue.

How should I display UTF-8 now?

Thank you.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin 2.6.0: unreadable UTF-8 in Windows console

[Brian Inglis]

On 2016-09-30 22:34, Brian Inglis wrote:
> On 2016-09-30 20:13, Ivan Vanyushkin wrote:
>> Something has changed in version 2.6.0, and now UTF-8 text can't be displayed in Windows console (cmd).
>> 1. Create a file "test.txt" with non-ASCII text in UTF-8 encoding.
>> 2. Run "cmd".
>> 3. Run:
>> C:\Cygwin\bin\cat test.txt
>> â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’ 8000 â–’â–’. â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’.
>> Non-ASCII text is not readable. Older Cygwin 2.5.2 has no such issue.
>> C:\Cygwin\bin\uname -a
>> CYGWIN_NT-10.0 PCName 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin
>> C:\Cygwin\bin\locale
>> LANG=
>> LC_CTYPE="C.UTF-8"
>> LC_NUMERIC="C.UTF-8"
>> LC_TIME="C.UTF-8"
>> LC_COLLATE="C.UTF-8"
>> LC_MONETARY="C.UTF-8"
>> LC_MESSAGES="C.UTF-8"
>> LC_ALL=
>> Same issue with any other commands like "grep", or with utilities built and run under Cygwin 2.6.0.
>> Same issue in other Windows consoles, like ConEmu or FAR Manager.
>> If I change Windows console encoding to UTF-8 (run: "chcp 65001"), file can be correctly displayed natively
>> (run: "type test.txt"), but Cygwin "cat" still has the same issue.
>> How should I display UTF-8 now?
>
> No problems here - same setup.
> Don't have files containing UTF-8 specials handy, but do have with Latin1 (ISO-8859-1) specials,
> convertable to UTF-8.
> Stripped common ASCII-only lines from output below.
> Default email encoding is Unicode (hopefully UTF-8) not Western (presumably Latin1), so should render accurately.
>
> $ uname -srvmo
> CYGWIN_NT-10.0 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin
> $ locale
> LANG=C.UTF-8
> LC_CTYPE="C.UTF-8"
> LC_NUMERIC="C.UTF-8"
> LC_TIME="C.UTF-8"
> LC_COLLATE="C.UTF-8"
> LC_MONETARY="C.UTF-8"
> LC_MESSAGES="C.UTF-8"
> LC_ALL=C.UTF-8
> $ egrep -a 'Deg|LF' latin1.txt # -a needed to override binary assumption - garbled characters
> DegN='â–’N'
> DegW='â–’W'
> Y2LF='%sâ–’%s %s %s'
> Y2LLF='|â–’%.0s|'
> LF='|â–’'.YFP.'|'
> $ iconv -f iso-8859-1 -t utf-8 latin1.txt | egrep 'Deg|LF' # good utf-8 characters
> DegN='°N'
> DegW='°W'
> Y2LF='%s±%s %s %s'
> Y2LLF='|±%.0s|'
> LF='|±'.YFP.'|'

Sorry - this was mintty - you used cmd!
Saw similar problems you had until I set LC_ALL=C.UTF-8 (and LANG for consistency, but doesn't really matter) and chcp 65001.
Then type and Cygwin commands produce the same output.
Without CP65001 (and a Unicode console font mapping most characters - I use DejaVu Sans Mono everywhere I can) there may be no valid encoding for UTF-8 special characters in your default console CP (437 for US, 850 for non-US, others for localized versions).
Unfortunately then less displays spaces as squares, so you may have to set PAGER=more for readability.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin 2.6.0: unreadable UTF-8 in Windows console

[Brian Inglis]

On 2016-09-30 22:34, Brian Inglis wrote:
> On 2016-09-30 20:13, Ivan Vanyushkin wrote:
>> Something has changed in version 2.6.0, and now UTF-8 text can't be displayed in Windows console (cmd).
>> 1. Create a file "test.txt" with non-ASCII text in UTF-8 encoding.
>> 2. Run "cmd".
>> 3. Run:
>> C:\Cygwin\bin\cat test.txt
>> â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’ 8000 â–’â–’. â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’ â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’.
>> Non-ASCII text is not readable. Older Cygwin 2.5.2 has no such issue.
>> C:\Cygwin\bin\uname -a
>> CYGWIN_NT-10.0 PCName 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin
>> C:\Cygwin\bin\locale
>> LANG=
>> LC_CTYPE="C.UTF-8"
>> LC_NUMERIC="C.UTF-8"
>> LC_TIME="C.UTF-8"
>> LC_COLLATE="C.UTF-8"
>> LC_MONETARY="C.UTF-8"
>> LC_MESSAGES="C.UTF-8"
>> LC_ALL=
>> Same issue with any other commands like "grep", or with utilities built and run under Cygwin 2.6.0.
>> Same issue in other Windows consoles, like ConEmu or FAR Manager.
>> If I change Windows console encoding to UTF-8 (run: "chcp 65001"), file can be correctly displayed natively
>> (run: "type test.txt"), but Cygwin "cat" still has the same issue.
>> How should I display UTF-8 now?
>
> No problems here - same setup.
> Don't have files containing UTF-8 specials handy, but do have with Latin1 (ISO-8859-1) specials,
> convertable to UTF-8.
> Stripped common ASCII-only lines from output below.
> Default email encoding is Unicode (hopefully UTF-8) not Western (presumably Latin1), so should render accurately.
>
> $ uname -srvmo
> CYGWIN_NT-10.0 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin
> $ locale
> LANG=C.UTF-8
> LC_CTYPE="C.UTF-8"
> LC_NUMERIC="C.UTF-8"
> LC_TIME="C.UTF-8"
> LC_COLLATE="C.UTF-8"
> LC_MONETARY="C.UTF-8"
> LC_MESSAGES="C.UTF-8"
> LC_ALL=C.UTF-8
> $ egrep -a 'Deg|LF' latin1.txt # -a needed to override binary assumption - garbled characters
> DegN='â–’N'
> DegW='â–’W'
> Y2LF='%sâ–’%s %s %s'
> Y2LLF='|â–’%.0s|'
> LF='|â–’'.YFP.'|'
> $ iconv -f iso-8859-1 -t utf-8 latin1.txt | egrep 'Deg|LF' # good utf-8 characters
> DegN='°N'
> DegW='°W'
> Y2LF='%s±%s %s %s'
> Y2LLF='|±%.0s|'
> LF='|±'.YFP.'|'

Sorry - this was mintty - you used cmd!
Saw similar problems you had until I set LC_ALL=C.UTF-8 (and LANG for consistency, but doesn't really matter) and chcp 65001.
Then type and Cygwin commands produce the same output.
Without CP65001 (and a Unicode console font mapping most characters - I use DejaVu Sans Mono everywhere I can) there may be no valid encoding for UTF-8 special characters in your default console CP (437 for US, 850 for non-US, others for localized versions).
Unfortunately then less displays spaces as squares, so you may have to set PAGER=more for readability.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin 2.6.0: unreadable UTF-8 in Windows console

[Ivan Vanyushkin]

"set LANG=C.UTF-8" has fixed the issue on Cygwin 2.6.0. But documentation says [1], that
"The default locale in the absence of the aforementioned locale environment variables is "C.UTF-8"."
Seems this is broken in Cygwin 2.6.0.

"chcp 65001", console font or console charset doesn't matter here.

This is bad, because now I can't share compiled binaries to anyone, because users will have
no LANG in environment variable, and any non-ACSII text will not be readable.
For example, list running Windows services:
sc query | grep -i "running"
- will not work for not-English Windows, because output in console will not be readable.
Watch Windows log:
tail -f C:\Windows\Logs\SomeLog.log
- will be not readable if there are some non-English file names.

I think locale should remain default UTF-8, as in Cygwin 2.5.2. This is expected by both
applications and users.

Tests:

// Run Windows console.
cmd

C:\Cygwin_2.6.0\bin\echo ±5°> utf-8.txt

C:\Cygwin_2.6.0\bin\od -t x1z utf-8.txt
0000000 c2 b1 35 c2 b0 0d 0a                             >..5....<
// We have UTF-8 now in "utf-8.txt" file.

C:\Cygwin_2.6.0\bin\locale
LANG=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

C:\Cygwin_2.6.0\bin\cat utf-8.txt
â–’â–’5â–’â–’

C:\Cygwin_2.6.0\bin\env LANG=C.UTF-8 C:\Cygwin_2.6.0\bin\cat utf-8.txt
±5°
// Fixed! But what is default locale then?

C:\Cygwin_2.6.0\bin\env LANG=C.CP1251 C:\Cygwin_2.6.0\bin\cat utf-8.txt
В±5В°

C:\Cygwin_2.6.0\bin\env LANG=C.CP866 C:\Cygwin_2.6.0\bin\cat utf-8.txt
┬▒5┬░

C:\Cygwin_2.6.0\bin\env LANG=C.ISO8859-1 C:\Cygwin_2.6.0\bin\cat utf-8.txt
±5°
// Doesn't match. I have no idea what is default locale in Cygwin 2.6.0.

// Let's try console native encoding
echo ±5°> cp866.txt

C:\Cygwin_2.6.0\bin\od -t x1z cp866.txt
0000000 2b 35 f8 0d 0a                                   >+5...<

type cp866.txt
+5°

C:\Cygwin_2.6.0\bin\cat cp866.txt
+5â–’
// Bad. Cygwin 2.6.0 can't display even non-UTF-8.


// Try filenames:
ls -al
lrwxrwxrwx  1 vanav                        â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’â–’       33 Sep 17 07:43 ''$'\320\234\320\276\320\270'' '$'\320\2
64\320\276\320\272\321\203\320\274\320\265\320\275\321\202\321\213' -> /cygdrive/c/Users/Vanav/Documents
// Bad.

C:\Cygwin_2.6.0\bin\env LANG=C.UTF-8 C:\Cygwin_2.6.0\bin\ls -al
lrwxrwxrwx  1 vanav          система             33 Sep 17 07:43 'Мои документы' -> /cygdrive/c/Users/Vanav/Documents
// Good.


// Now try previous Cygwin 2.5.2.
C:\Cygwin_2.5.2\bin\locale
LANG=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

C:\Cygwin_2.5.2\bin\cat utf-8.txt
±5°
// Good.

C:\Cygwin_2.5.2\bin\env LANG=C.UTF-8 C:\Cygwin_2.5.2\bin\cat utf-8.txt
±5°
// Good.


[1] https://cygwin.com/cygwin-ug-net/setup-locale.html



Saturday, October 1, 2016, 8:15:02 AM, you wrote:

> On 2016-09-30 22:34, Brian Inglis wrote:
> Sorry - this was mintty - you used cmd!
> Saw similar problems you had until I set LC_ALL=C.UTF-8 (and LANG
> for consistency, but doesn't really matter) and chcp 65001.
> Then type and Cygwin commands produce the same output.
> Without CP65001 (and a Unicode console font mapping most characters
> - I use DejaVu Sans Mono everywhere I can) there may be no valid
> encoding for UTF-8 special characters in your default console CP
> (437 for US, 850 for non-US, others for localized versions).
> Unfortunately then less displays spaces as squares, so you may have
> to set PAGER=more for readability.



-- 
Best regards,
 Ivan                            mailto:vanav@vanav.org


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin 2.6.0: unreadable UTF-8 in Windows console

[Bengt Larsson]

Ivan Vanyushkin wrote:
>Something has changed in version 2.6.0, and now UTF-8 text can't be displayed in Windows console (cmd).
>
>1. Create a file "test.txt" with non-ASCII text in UTF-8 encoding.
>2. Run "cmd".
>3. Run:
>
>C:\Cygwin\bin\cat test.txt
>???????????????? ?????????????? ???? ?????? 8000 ??. ???? ?????????????????????? ??????????.
>
>Non-ASCII text is not readable. Older Cygwin 2.5.2 has no such issue.
>
>C:\Cygwin\bin\uname -a
>CYGWIN_NT-10.0 PCName 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin
>
>C:\Cygwin\bin\locale
>LANG=
>LC_CTYPE="C.UTF-8"
>LC_NUMERIC="C.UTF-8"
>LC_TIME="C.UTF-8"
>LC_COLLATE="C.UTF-8"
>LC_MONETARY="C.UTF-8"
>LC_MESSAGES="C.UTF-8"
>LC_ALL=

You don't have LANG set to "C.UTF-8". Do that.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin 2.6.0: unreadable UTF-8 in Windows console

[Ivan Vanyushkin]

I want to share binary built under Cygwin 2.6.0 with other user, that has no LANG set.
In previous version all binaries worked correctly with UTF-8 input text.
But now this doesn't work as expected.

Some more simple tests.

// Run Windows console.
cmd

C:\Cygwin_2.6.0\bin\echo ±5°
â–’â–’5â–’â–’

C:\Cygwin_2.6.0\bin\echo ±5°|C:\Cygwin_2.6.0\bin\grep .
â–’â–’5â–’â–’
// Bad in Cywgin 2.6.0. Maybe this is UTF-8? See below.

// Now try in previous version.
C:\Cygwin_2.5.2\bin\echo ±5°
±5°

C:\Cygwin_2.5.2\bin\echo ±5°|C:\Cygwin_2.5.2\bin\grep .
±5°
// Good in Cygwin 2.5.2.


// Looks like changing console to UTF-8 codepage may fix the issue?
chcp 65001
Active code page: 65001

C:\Cygwin_2.6.0\bin\echo ±5°
â–’â–’5â–’â–’

C:\Cygwin_2.6.0\bin\echo ±5°|C:\Cygwin_2.6.0\bin\grep .
â–’â–’5â–’â–’

echo ±5°
±5°

echo ±5°|C:\Cygwin_2.6.0\bin\grep .
â–’â–’5â–’â–’
// Bad in Cygwin 2.6.0. Seems this is not UTF-8, just broken text.

// Now try in previous version.
C:\Cygwin_2.5.2\bin\echo ±5°
±5°

C:\Cygwin_2.5.2\bin\echo ±5°|C:\Cygwin_2.5.2\bin\grep .
±5°

echo ±5°
±5°

echo ±5°|C:\Cygwin_2.5.2\bin\grep .
±5°
// Works good in Cygwin 2.5.2 even with native "echo", because it produces UTF-8 now.


Sunday, October 2, 2016, 9:29:11 AM, you wrote:
> You don't have LANG set to "C.UTF-8". Do that.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin 2.6.0: unreadable UTF-8 in Windows console

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1705 bytes --]

On Oct  1 05:13, Ivan Vanyushkin wrote:
> Something has changed in version 2.6.0, and now UTF-8 text can't be displayed in Windows console (cmd).
> 
> 1. Create a file "test.txt" with non-ASCII text in UTF-8 encoding.
> 2. Run "cmd".
> 3. Run:
> 
> C:\Cygwin\bin\cat test.txt
> ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒ ▒▒▒▒▒▒ 8000 ▒▒. ▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒▒▒.
> 
> Non-ASCII text is not readable. Older Cygwin 2.5.2 has no such issue.
> 
> C:\Cygwin\bin\uname -a
> CYGWIN_NT-10.0 PCName 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin
> 
> C:\Cygwin\bin\locale
> LANG=
> LC_CTYPE="C.UTF-8"
> LC_NUMERIC="C.UTF-8"
> LC_TIME="C.UTF-8"
> LC_COLLATE="C.UTF-8"
> LC_MONETARY="C.UTF-8"
> LC_MESSAGES="C.UTF-8"
> LC_ALL=
> 
> Same issue with any other commands like "grep", or with utilities built and run under Cygwin 2.6.0.
> Same issue in other Windows consoles, like ConEmu or FAR Manager.
> If I change Windows console encoding to UTF-8 (run: "chcp 65001"), file can be correctly displayed natively
> (run: "type test.txt"), but Cygwin "cat" still has the same issue.
> 
> How should I display UTF-8 now?

I applied a patch to fix this regression and uploaded a developer
snapshot with this change to https://cygwin.com/snapshots/

Please test.  Just exchanging the Cygwin DLL is sufficient, no need to
install the entire package.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Cygwin 2.6.0: unreadable UTF-8 in Windows console

[Ivan Vanyushkin]

Hello Corinna,

Wednesday, October 19, 2016, 2:45:16 PM, you wrote:
> I applied a patch to fix this regression and uploaded a developer
> snapshot with this change to https://cygwin.com/snapshots/

> Please test.  Just exchanging the Cygwin DLL is sufficient, no need to
> install the entire package.

Looks good with new DLL, thank you!


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple