[bug] globify dospath reacts poorly with escaped double quotes

[bug] globify dospath reacts poorly with escaped double quotes

[Mingye Wang]

Hi,

This bug is inherited from early versions of Cygwin. It's so old that
MSYS2 has this problem too.

There is no way of conveying a double quote in an argument once
globify() decides it has seen a dospath. Neither the `\"` nor `""`
work, because they are both unified to `\"` in quoted() and turned
into a `\\` pattern in globify().

This is problematic for programmers trying to write a routine to
reliably escape an argument for the Cygwin command-line.

A way to patch the problem is with a lookahead in globify():

if (dos_spec && *s == '\\') {
/**/p++ = '\\';
/**/if (s[1] == '"' && s[2]) {
/****/*p = *++s;
/****/continue;
/**/}
}
*p = *s;

[Apologies for the formatting; the gmail web editor hates leading spaces.]

(Note: The backslash thing has always been different from the MSCRT
handling, which only transforms backslashes followed by a double
quote. But this is fine as long as we are internally consistent.
Well... is it documented anywhere?)

--
Regards,
Mingye Wang

--
Mingye Wang (Artoria2e5)

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug] globify dospath reacts poorly with escaped double quotes

[Brian Inglis]

On 2019-10-08 03:05, Mingye Wang wrote:
> This bug is inherited from early versions of Cygwin. It's so old that
> MSYS2 has this problem too.

Probably not a bug then but a feature for Cygwin.
Msys2 is another system with different goals, using GNU toolchains to build
native Windows programs, not a platform for running POSIX applications like
Cygwin, and OT for this list.

> There is no way of conveying a double quote in an argument once
> globify() decides it has seen a dospath. Neither the `\"` nor `""`
> work, because they are both unified to `\"` in quoted() and turned
> into a `\\` pattern in globify().

Msys2 tools have to make their own arrangements if they support native Windows
paths.
Personally I found when I used to use DOS and Windows tools, it was easier using
slashes instead of backslashes as directory separators, as most interfaces did
not care, including most DOS & Windows APIs.

> This is problematic for programmers trying to write a routine to
> reliably escape an argument for the Cygwin command-line.

Backslash escaping and switching enclosing quotes on shell command lines works
reliably to pass any arguments into Cygwin programs, as do the various shell
command line parameter and wildcard expansions.
Passing special characters into arguments interpreted by other programs requires
additional care.

> A way to patch the problem is with a lookahead in globify():
> 
> if (dos_spec && *s == '\\') {
> /**/p++ = '\\';
> /**/if (s[1] == '"' && s[2]) {
> /****/*p = *++s;
> /****/continue;
> /**/}
> }
> *p = *s;
> 
> [Apologies for the formatting; the gmail web editor hates leading spaces.]
> 
> (Note: The backslash thing has always been different from the MSCRT
> handling, which only transforms backslashes followed by a double
> quote. But this is fine as long as we are internally consistent.
> Well... is it documented anywhere?)

Support of DOS paths is inconsistent in Cygwin utilities and may not work: use
cygpath, or the low level API, to convert DOS to POSIX paths before passing to
Cygwin programs, or functions.

Backslash should only be used to escape command line characters with special
meaning to the shell, or escapes in strings in other languages.

Any other use should specify what kinds or arguments ypu are trying to handle,
how you are getting your arguments in, and passing them to globify.

Invoking Cygwin programs from other Cygwin programs is best done using the exec
or spawn functions with (unescaped, unquoted) arguments in varargs arg lists or
arrays.

Invoking Windows programs works best when done from a cmd wrapper, but anything
involving any Windows command line requires work to generalize.
See previous recent posts for what is required.

Avoid system() and similar calls if possible, as they will then go thru an
additional shell layer.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug] globify dospath reacts poorly with escaped double quotes

[Ken Brown]

On 10/8/2019 5:05 AM, Mingye Wang wrote:
> Hi,
> 
> This bug is inherited from early versions of Cygwin. It's so old that
> MSYS2 has this problem too.
> 
> There is no way of conveying a double quote in an argument once
> globify() decides it has seen a dospath. Neither the `\"` nor `""`
> work, because they are both unified to `\"` in quoted() and turned
> into a `\\` pattern in globify().
> 
> This is problematic for programmers trying to write a routine to
> reliably escape an argument for the Cygwin command-line.
> 
> A way to patch the problem is with a lookahead in globify():
> 
> if (dos_spec && *s == '\\') {
> /**/p++ = '\\';
> /**/if (s[1] == '"' && s[2]) {
> /****/*p = *++s;
> /****/continue;
> /**/}
> }
> *p = *s;
> 
> [Apologies for the formatting; the gmail web editor hates leading spaces.]
> 
> (Note: The backslash thing has always been different from the MSCRT
> handling, which only transforms backslashes followed by a double
> quote. But this is fine as long as we are internally consistent.
> Well... is it documented anywhere?)

I don't know enough about Windows command-line quoting to comment on the 
substance of this.  But it would help those who do if you would send your patch 
to the cygwin-patches mailing list using git format-patch and git send-email. 
And please use the formatting conventions of the surrounding code.  For example,

if (dos_spec && *s == '\\')
   {
     [...]
   }

Thanks.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug] globify dospath reacts poorly with escaped double quotes

[Brian Inglis]

On 2019-10-10 02:30, Mingye Wang wrote:
> On Tue, 8 Oct 2019 14:09:18 -0600, Brian Inglis wrote:
>> On 2019-10-08 03:05, Mingye Wang wrote:
>> >> This bug is inherited from early versions of Cygwin. It's so old that
>>> MSYS2 has this problem too.
>> 
>> Probably not a bug then but a feature for Cygwin.
>> Msys2 is another system with different goals, using GNU toolchains to build
>> native Windows programs, not a platform for running POSIX applications like
>> Cygwin, and OT for this list.
>> 
>>> There is no way of conveying a double quote in an argument once
>>> globify() decides it has seen a dospath. Neither the `\"` nor `""`
>>> work, because they are both unified to `\"` in quoted() and turned
>>> into a `\\` pattern in globify().
>> 
>> Msys2 tools have to make their own arrangements if they support native Windows
>> paths.
>> Personally I found when I used to use DOS and Windows tools, it was easier using
>> slashes instead of backslashes as directory separators, as most interfaces did
>> not care, including most DOS & Windows APIs.

[please Reply to List, or Reply to All and send only to the mailing list
address, to maintain email thread references]

> Use the source, Luke. Globbing on the cmdline, as we all know, is something
> you only care about when you are *not* passing in arguments from a Unix-type
> cmdline, and build_argv in dcrt0.cc does *only* run globify if the parent
> process is *not* Cygwin-powered.
We need to understand your use case(s) and issue(s), and you to explain those to
us.
Please provide at least test vectors showing what you are trying to get, what
you are providing, what you are getting, the difference, and why you think it
should be different.

You can often get code formatted for web mail by using gist or similar editors
or other markdown environments and paste the text as formatted into the web mail
compose window.

You can also attach as text a file or extract showing the file(s) and context(s)
where you create the content, and/or a simple test script or program to do so,
and/or where you think the problem resides, or a patch as at least a context
diff, preferably git format-patch or send-email output.

If you send a patch, please also show how your test vectors, plus a variety of
more normal usage, work before and after your patch is applied.

>>> This is problematic for programmers trying to write a routine to
>>> reliably escape an argument for the Cygwin command-line.
>> 
>> Backslash escaping and switching enclosing quotes on shell command lines works
>> reliably to pass any arguments into Cygwin programs, as do the various shell
>> command line parameter and wildcard expansions.

> See above.

Where?
Have you considered bypassing command lines using sockets, pipes, shared memory
segments, memory mapped temporary files, etc.

Have you considered trying to pass argument and environment vectors directly to
the Cygwin program, as would be done by an invoking Cygwin program?

Have you tried using single quotes around problematic arguments, quoted as
required to pass this thru from the calling environment, which if you're calling
a program directly, should not require any escaping or quoting, or could include
whatever other escaping or quoting you need to pass the info transparently,
perhaps via an interop wrapper program or script?

>> Passing special characters into arguments interpreted by other programs requires
>> additional care.
>> 
>>> A way to patch the problem is with a lookahead in globify():
>>>
>>> if (dos_spec && *s == '\\') {
>>> /**/p++ = '\\';
>>> /**/if (s[1] == '"' && s[2]) {
>>> /****/*p = *++s;
>>> /****/continue;
>>> /**/}
>>> }
>>> *p = *s;
>>>
>>> [Apologies for the formatting; the gmail web editor hates leading spaces.]
>>>
>>> (Note: The backslash thing has always been different from the MSCRT
>>> handling, which only transforms backslashes followed by a double
>>> quote. But this is fine as long as we are internally consistent.
>>> Well... is it documented anywhere?)
>> 
>> Support of DOS paths is inconsistent in Cygwin utilities and may not work: use
>> cygpath, or the low level API, to convert DOS to POSIX paths before passing to
>> Cygwin programs, or functions.
>> 
>> Backslash should only be used to escape command line characters with special
>> meaning to the shell, or escapes in strings in other languages.
>> 
>> Any other use should specify what kinds or arguments ypu are trying to handle,
>> how you are getting your arguments in, and passing them to globify.
>> 
>> Invoking Cygwin programs from other Cygwin programs is best done using the exec
>> or spawn functions with (unescaped, unquoted) arguments in varargs arg lists or
>> arrays.
>> 
>> Invoking Windows programs works best when done from a cmd wrapper, but anything
>> involving any Windows command line requires work to generalize.
>> See previous recent posts for what is required.
>> 
>> Avoid system() and similar calls if possible, as they will then go thru an
>> additional shell layer.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug] globify dospath reacts poorly with escaped double quotes

[Mingye Wang]

On Tue, 8 Oct 2019 14:09:18 -0600, Brian Inglis wrote:

> Msys2 tools have to make their own arrangements if they support native Windows
> paths.
> Personally I found when I used to use DOS and Windows tools, it was easier using
> slashes instead of backslashes as directory separators, as most interfaces did
> not care, including most DOS & Windows APIs.

Use the source, Luke. Globbing on the cmdline, as we all know, is
something you only
care about when you are *not* passing in arguments from a Unix-type cmdline, and
build_argv in dcrt0.cc does *only* run globify if the parent process is *not*
Cygwin-powered.

> Backslash escaping and switching enclosing quotes on shell command lines...

See above.

--
Mingye Wang (Artoria2e5)

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple