cmd.exe and file name conversion from comman line argument

cmd.exe and file name conversion from comman line argument

[h_ono]

Hello,

There is strange behavior when naming a file from command line argument in 
cmd.exe cosole,
using touch or mv.

I use:
touch (GNU coreutils) 8.26
Packaged by Cygwin (8.26-2)

cygwin1.dll version 2.10.0

in cmd.exe, doing

> touch aaa
and
> touch 'aaa'

give same file name aaa.

> ls
 aaa

but, when I use Japanese characters as file name, it gives different file 
names
with and without the quote.

> env LANG=ja_JP.UTF-8 touch あああ

gives

> env LANG=ja_JP.UTF-8 ls
  あああ

but,

> env LANG=ja_JP.UTF-8 touch 'あああ'

gives file name with quote not stripped.

> env LANG=ja_JP.UTF-8 ls
  'あああ'

If called from mintty + bash,

$ env LANG=ja_JP.UTF-8 touch 'あああ'

gives

$ env LANG=ja_JP.UTF-8 ls
  あああ

The problem is when I create a file containing spaces in its name.
Creating a file in cmd.exe like:

> env LANG=ja_JP.UTF-8 touch 'ああ あ'

gives

> env LANG=ja_JP.UTF-8 ls
"'ああ あ'"

(which is a file named 'ああ あ', not ああ あ)
It seems, there is no way to create a file containing a space in its name, 
like
 ああ あ
with commands such as touch or mv.

Is it a normal behavior, or a bug?
Is there any way to create a file containing Japanese characters and 
spaces in its name
without quote added?

FYI, With old cygwin (cygwin1.dll version 1.5.25), there were not such 
problem.
Even from cmd.exe,

> touch 'ああ あ'

gave a file named 'ああ あ' (without the quote in real.)
----------
Hiroo Ono

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cmd.exe and file name conversion from comman line argument

[Marco Atzeri]

Am 27.07.2018 um 02:56 schrieb h_ono@ot.olympus.co.jp:
> Hello,
>
> There is strange behavior when naming a file from command line argument in
> cmd.exe cosole,
> using touch or mv.
>
> I use:
> touch (GNU coreutils) 8.26
> Packaged by Cygwin (8.26-2)
>
> cygwin1.dll version 2.10.0
>
> in cmd.exe, doing

what happen if you use bash ?

Regards
Marco

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cmd.exe and file name conversion from comman line argument

[Andrey Repin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 2399 bytes --]

Greetings, h_ono@ot.olympus.co.jp!

> Hello,

> There is strange behavior when naming a file from command line argument in 
> cmd.exe cosole,
> using touch or mv.

No, your comparison is not fair.

> I use:
> touch (GNU coreutils) 8.26
> Packaged by Cygwin (8.26-2)

You're using "env" to begin with, which alters processing of parameters.
But I can't reproduce your case. The result is definitely does not contain the quotes.

chcp 65001
set LANG=ru_RU.UTF-8
env LANG=ja_JP.UTF-8 touch 'あああ'

It creates exactly the name I'd expect, even though I can't see it in console.
But Explorer don't lie and shows it proper.

> cygwin1.dll version 2.10.0

> in cmd.exe, doing

>> touch aaa
> and
>> touch 'aaa'

> give same file name aaa.

>> ls
>  aaa

> but, when I use Japanese characters as file name, it gives different file 
> names
> with and without the quote.

>> env LANG=ja_JP.UTF-8 touch あああ

> gives

>> env LANG=ja_JP.UTF-8 ls
>   あああ

> but,

>> env LANG=ja_JP.UTF-8 touch 'あああ'

> gives file name with quote not stripped.

>> env LANG=ja_JP.UTF-8 ls
>   'あああ'

> If called from mintty + bash,

> $ env LANG=ja_JP.UTF-8 touch 'あああ'

> gives

> $ env LANG=ja_JP.UTF-8 ls
>   あああ

> The problem is when I create a file containing spaces in its name.
> Creating a file in cmd.exe like:

>> env LANG=ja_JP.UTF-8 touch 'ああ あ'

> gives

>> env LANG=ja_JP.UTF-8 ls
> "'ああ あ'"

> (which is a file named 'ああ あ', not ああ あ)
> It seems, there is no way to create a file containing a space in its name, 
> like
>  ああ あ
> with commands such as touch or mv.

> Is it a normal behavior, or a bug?
> Is there any way to create a file containing Japanese characters and 
> spaces in its name
> without quote added?

> FYI, With old cygwin (cygwin1.dll version 1.5.25), there were not such 
> problem.
> Even from cmd.exe,

>> touch 'ああ あ'

> gave a file named 'ああ あ' (without the quote in real.)


-- 
With best regards,
Andrey Repin
Friday, July 27, 2018 7:50:49

Sorry for my terrible english...\x03B‹KCB”\x1c›Ø›\x19[H\x1c™\^[ܝ\x1cΈ\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÜ\x1c›Ø›\x19[\Ëš\x1d^[[\x03B‘TNˆ\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÙ˜\KÃB‘^[ØÝ[Y[\x18]\x1a[ÛŽˆ\b\b\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÙ^[ØÜËš\x1d^[[\x03B•[œÝXœØÜšX™H\x1a[™›Îˆ\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÛ[\vÈÝ[œÝXœØÜšX™K\Ú[\^[\x19CBƒB

Re: cmd.exe and file name conversion from comman line argument

[Brian Inglis]

On 2018-07-26 18:56, h_ono@ot.olympus.co.jp wrote:
> There is strange behavior when naming a file from command line argument in 
> cmd.exe console, using touch or mv.
> I use:
> touch (GNU coreutils) 8.26
> Packaged by Cygwin (8.26-2)
> cygwin1.dll version 2.10.0
> in cmd.exe, doing
>> touch aaa
> and
>> touch 'aaa'
> give same file name aaa.
>> ls
>  aaa
> but, when I use Japanese characters as file name, it gives different file 
> names with and without the quote.
>> env LANG=ja_JP.UTF-8 touch あああ
> gives
>> env LANG=ja_JP.UTF-8 ls
>   あああ
> but,
>> env LANG=ja_JP.UTF-8 touch 'あああ'
> gives file name with quote not stripped.
>> env LANG=ja_JP.UTF-8 ls
>   'あああ'
> If called from mintty + bash,
> $ env LANG=ja_JP.UTF-8 touch 'あああ'
> gives
> $ env LANG=ja_JP.UTF-8 ls
>   あああ
> The problem is when I create a file containing spaces in its name.
> Creating a file in cmd.exe like:
>> env LANG=ja_JP.UTF-8 touch 'ああ あ'
> gives
>> env LANG=ja_JP.UTF-8 ls
> "'ああ あ'"
> (which is a file named 'ああ あ', not ああ あ)
> It seems, there is no way to create a file containing a space in its name, 
> like
>  ああ あ
> with commands such as touch or mv.
> Is it a normal behavior, or a bug?
> Is there any way to create a file containing Japanese characters and spaces
> in its name without quote added?
> FYI, With old cygwin (cygwin1.dll version 1.5.25), there were not such 
> problem. Even from cmd.exe,
>> touch 'ああ あ'
> gave a file named 'ああ あ' (without the quote in real.)

Under a Unix shell, both double quote " and single quote/apostrophe ' are
quoting metacharacters, as is backquote/grave `, but any character may be used
in a file name by prefixing with the escape character \.
Under Windows cmd only double quote " is a quoting metacharacter; path spec
delimiters : \ /, wild card characters * ?, and redirection characters | < > are
also not allowed in file names. So to create a Windows file name with a space
from cmd, use only double quotes " around the name.
Under Cygwin, any file name characters disallowed by Windows are mapped into
Unicode private use area characters, but converted back on display, so under
Windows programs those files' long names will display characters without
assigned glyphs.

Note that in ls, nongraphic characters in a file name entry e.g. space, causes
that file name entry to be displayed single quoted 'a b'; specifying ls
-N|--literal|--quoting-style=literal omits the single quotes, but nongraphic
characters are displayed on terminals as question marks ?, unless
--show-control-chars is also specified.
Run "info ls 'Formatting the file names'" to see full explanations.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cmd.exe and file name conversion from comman line argument

[h_ono]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 2750 bytes --]

Thank you

> From: Marco Atzeri <marco.atzeri@gmail.com>

> what happen if you use bash ?

With cmd + bash, the problem does not occur, but I would like to use cmd 
without bash, if possible.


> From: Brian Inglis <Brian.Inglis@SystematicSw.ab.ca>
> To: cygwin@cygwin.com
> Date: 2018/07/27 14:25
> Subject: Re: cmd.exe and file name conversion from comman line argument
> Sent by: cygwin-owner@cygwin.com

> Under a Unix shell, both double quote " and single quote/apostrophe ' 
are
> quoting metacharacters, as is backquote/grave `, but any character may 
be used
> in a file name by prefixing with the escape character \.
> Under Windows cmd only double quote " is a quoting metacharacter; path 
spec
> delimiters : \ /, wild card characters * ?, and redirection 
> characters | < > are
> also not allowed in file names. So to create a Windows file name with a 
space
> from cmd, use only double quotes " around the name.

Thank you. quoting only the space character worked, which seems to solve
my problem.

from cmd, with LANG environment variables set to ja_JP.UTF-8

c:\cygwin\home\hiroo> touch ああ" "あ

or

c:\cygwin\home\hiroo> touch ああ' 'あ

gives a file named ああ あ.

The rest may be superfluous, but just to compare with.
Quoting Japanese characters too did not work.

c:\cygwin\home\hiroo> touch "あああ"

gives

c:\cygwin\home\hiroo> ls
'"あああ"'

(and shown as ・あああ・ in explorer)

while

c:\cygwin\home\hiroo> touch "aaa"

gives

c:\cygwin\home\hiroo> ls
aaa

Doing

c:\cygwin\home\hiroo> echo > "あああ"

(in this case, the file name is surely processed by cmd)
gives a file named あああ (without quotes).


> Under Cygwin, any file name characters disallowed by Windows are mapped 
into
> Unicode private use area characters, but converted back on display, so 
under
> Windows programs those files' long names will display characters without
> assigned glyphs.
> 
> Note that in ls, nongraphic characters in a file name entry e.g. space, 
causes
> that file name entry to be displayed single quoted 'a b'; specifying ls
> -N|--literal|--quoting-style=literal omits the single quotes, but 
nongraphic
> characters are displayed on terminals as question marks ?, unless
> --show-control-chars is also specified.
> Run "info ls 'Formatting the file names'" to see full explanations.
> 
> -- 
> Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
> 

---------
Hiroo Ono
\x03B‹KCB”\x1c›Ø›\x19[H\x1c™\^[ܝ\x1cΈ\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÜ\x1c›Ø›\x19[\Ëš\x1d^[[\x03B‘TNˆ\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÙ˜\KÃB‘^[ØÝ[Y[\x18]\x1a[ÛŽˆ\b\b\b\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÙ^[ØÜËš\x1d^[[\x03B•[œÝXœØÜšX™H\x1a[™›Îˆ\b\b\b\b\b\x1a\x1d\x1d\x1c\x0e‹ËØÞYÝÚ[‹˜ÛÛKÛ[\vÈÝ[œÝXœØÜšX™K\Ú[\^[\x19CBƒB

Re: cmd.exe and file name conversion from comman line argument

[h_ono]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="UTF-8", Size: 1291 bytes --]

> From: Hiroo Ono/jp/olympus@olympus
> To: cygwin@cygwin.com
> Date: 2018/07/27 15:57
> Subject: Re: cmd.exe and file name conversion from comman line argument
> Sent by: cygwin-owner@cygwin.com
> 

> c:\cygwin\home\hiroo> touch "あああ"
> 
> gives
> 
> c:\cygwin\home\hiroo> ls
> '"あああ"'

It may be related to build_argv() in 
https://github.com/mirror/newlib-cygwin/blob/master/winsup/cygwin/dcrt0.cc

line 320 to 328:
            /* Skip over characters until the closing quote */
            {
              sawquote = cmd;
              /* Handle quoting.  Only strip off quotes if the parent is
                 a Cygwin process, or if the word starts with a '@'.
                 In this case, the insert_file function needs an unquoted
                 DOS filename and globbing isn't performed anyway. */
              cmd = quoted (cmd, winshell && argc > 0 && *word != '@');
            }

As the comment says,

c:\cygwin\home\hiroo> touch @"あああ"

gives file named @あああ.
 
---------
Hiroo Ono

\0ТÒÐÐ¥\a&ö&ÆVÒ\a&W\x06÷'G3¢\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒ÷\a&ö&ÆV×2æ‡FÖÀФd\x15\x13¢\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöf\x17\x12ðФFö7VÖVçF\x17F–öã¢\x02\x02\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöFö72æ‡FÖÀÐ¥Vç7V'67&–&R\x06–æfó¢\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöÖÂò7Vç7V'67&–&R×6–×\x06ÆPРÐ

Re: cmd.exe and file name conversion from comman line argument

[Takashi Yano]

On Fri, 27 Jul 2018 15:56:39 +0900
> The rest may be superfluous, but just to compare with.
> Quoting Japanese characters too did not work.
> 
> c:\cygwin\home\hiroo> touch "あああ"
> 
> gives
> 
> c:\cygwin\home\hiroo> ls
> '"あああ"'

If you set LANG prior to touch using set instead of env,
it seems to behave as you expected.

C:\cygwin\home\yano\testdir>set LANG=ja_JP.UTF-8

C:\cygwin\home\yano\testdir>touch "あああ"

C:\cygwin\home\yano\testdir>ls
あああ

C:\cygwin\home\yano\testdir>dir
 Volume in drive C has no label.
 Volume Serial Number is 1E17-C0B1

 Directory of C:\cygwin\home\yano\testdir

2018/07/27  19:06    <DIR>          .
2018/07/27  19:06    <DIR>          ..
2018/07/27  19:06                 0 あああ
               1 File(s)              0 bytes
               2 Dir(s)  89,141,444,608 bytes free


-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cmd.exe and file name conversion from comman line argument

[h_ono]

hello,

> From: Takashi Yano <takashi.yano@nifty.ne.jp>
> To: cygwin@cygwin.com
> Date: 2018/07/27 19:16
> Subject: Re: cmd.exe and file name conversion from comman line argument
> Sent by: cygwin-owner@cygwin.com

> If you set LANG prior to touch using set instead of env,
> it seems to behave as you expected.
> 
> C:\cygwin\home\yano\testdir>set LANG=ja_JP.UTF-8
> 
> C:\cygwin\home\yano\testdir>touch "あああ"
> 
> C:\cygwin\home\yano\testdir>ls
> あああ

Confirmed. Thank you.
I thought using env and setting environment variable were identical. 
Sorry.
----------
Hiroo Ono

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple