Re: Need help with multibyte UTF-8 characters

[Doug Henderson <djndnbvg@gmail.com>]

On 11 December 2017 at 16:36, Thomas Taylor wrote:
> Thank you for your advice on setting my locale to en_US.UTF-8.
> Unfortunately, Cygwin still seems to have trouble displaying some three-byte
> UTF-8 encoded characters correctly.  For example, see the following snippet
> from a "sed" file.  This file attempts to convert XML-encoded filenames to
> UTF-8.  As you can see, it converts one- and two-byte encodings correctly,
> but fails on some three-byte encodings (the en dash, the em dash, and the
> ellipsis, all of which are displayed as a filled-in rectangle):
>

Your sed script works for me. I copy/pasted your sample script into
"cvt_script.sed" and also into "cvt_input.txt". My sed command looks
like: "sed --file=cvt_script.sed < cvt_input.txt > cvt_output.txt". It
correctly translates all the encoded utf-8 strings.

Your display may appear different if you are using different fonts in
mintty or the windows console. I am using Lucinda Console, 10pt and
Consolas 16, respectively. They display different glyphs for the
non-breaking space, but are otherwise identical. In mintty, I have
LANG and all the LC_* variables set to en_CA.UTF-8, and in the windows
console, to en_US.UTF-8.

I am running Win 10 and cygwin setup was last updated a couple or
three days ago.

Check the output of the "locale" command. All variables should have
the same value.

Is your cygwin installation up to date, or fairly close to current?
What wiindows version are you using?

HTH,
Doug


-- 
Doug Henderson, Calgary, Alberta, Canada - from gmail.com

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple