Re: Removing ^X in paths

[Brian Inglis <Brian.Inglis@SystematicSw.ab.ca>]

On 2022-02-02 21:12, Dennis Heimbigner wrote:
> On 2/2/2022 7:23 PM, L A Walsh wrote:
>> On 2022/02/02 12:40, Dennis Heimbigner wrote:
>>> It appears that windows now supports the UTF-8 codepage.
>> It has since early 2000's.
>>> I light of this, it seems time to change cygwin so it no longer adds 
>>> those
>>> control-x (^X)  characters in e.g. path names.
>> ^x is ASCII.  Cygwin doesn't insert ^X characters in paths.
>> Perhaps you are thinking of '\' which looks like ￥ (a capital 'Y' 
>> with 2 horizontal lines, (Fullwidth Yen Sign  U+FFE5)...if that's the 
>> case, some 8-bit font
>> displayed that sign instead of a backslash in non-unicode locals.
>> Are you using a 32-bit or 64-bit version of Cygwin?  on what version 
>> of windows?
>> If you still use a 32-bit version, you might need to move to a 64-bit 
>> version.
>> I know the 32-bit version sometimes had the problem because it supported
>> fewer fonts and fewer characters at the same time.
>> You might check out your locale (if in english, try setting:
>> LC_CTYPE="en_US.UTF-8"
>> in your shell and also check that your used font has a backslash in the
>> 0x7f position.
>> But in shell, ^x is usually a character to erase the whole line -- so 
>> it really
>> wouldn't do to have it in a PATH.
>> Hope this helps, and sorry if this is completely off base.

 > I am using 64bit.
 > And it has nothing to do misreading characters.
 > The ^X is described in this document:
 > https://www.cygwin.com/cygwin-ug-net/using-specialnames.html,
 > There you will see this text:
 > "If you don't want or can't use UTF-8 as character set for
 > whatever reason, you will nevertheless be able to access the
 > file. How does that work? When Cygwin converts the filename from
 > UTF-16 to your character set, it recognizes characters which
 > can't be converted. If that occurs, Cygwin replaces the
 > non-convertible character with a special character sequence. The
 > sequence starts with an ASCII CAN character (hex code 0x18,
 > equivalent Control-X), followed by the UTF-8 representation of
 > the character. The result is a filename containing some ugly
 > looking characters. While it doesn't look nice, it is nice,
 > because Cygwin knows how to convert this filename back to
 > UTF-16. The filename will be converted using your usual
 > character set. However, when Cygwin recognizes an ASCII CAN
 > character, it skips over the ASCII CAN and handles the following
 > bytes as a UTF-8 character. Thus, the filename is symmetrically
 > converted back to UTF-16 and you can access the file."
 > There is no obvious good reason to continue this convention.

This is not a convention, it is an interoperability feature, to allow 
unsupported characters to be used in filenames, otherwise Cygwin would 
have to fail the file open in locales where those characters are 
unsupported.

I have always used ASCII, ISO-8859-1/15, or UTF-8 and have never seen a 
^X in any filename, although I have produced many other control and 
special characters in filenames by error. ;^>

If you never use a limited character set locale with filenames using 
extended character sets you will never see this either.

This feature is for those who may be importing files with names in 
extended character sets but their selected locale only supports a 
limited character set.

Some users and nationalities still prefer to use locales with limited 
character sets, perhaps because their important apps still use them, and 
they are familiar with the related keyboard mappings and font glyphs.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]