Extra CR symbol from backticks on Cygwin 2.9.0

Extra CR symbol from backticks on Cygwin 2.9.0

[Nikolay Melekhin]

Command line utilities in Windows add CRLF as new line symbol in an output.

For example:

echo "Windows"
TEST=`cmd /c "echo Win"`
echo " ${TEST} " | od -t x1
echo "CYGWIN"
TEST=`echo Win`
echo " ${TEST} " | od -t x1

Output:

Windows
0000000 20 57 69 6e 0d 20 0a
0000007
CYGWIN
0000000 20 57 69 6e 20 0a
0000006

I'd expect that Cygwin must remove CRLF symbols from backtick results,
but remove only LF symbol. It gets different results on Windows and
Linux. Result on Windows has additional CR symbol and for example
string comparision, which is working in Linux, fails in Windows.

Is any kind of an option/setting which removes such CRs or it is a bug
in Cygwin?


Best regards,
Nikolay Melekhin

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Extra CR symbol from backticks on Cygwin 2.9.0

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 1281 bytes --]

On 09/11/2017 04:27 PM, Nikolay Melekhin wrote:
> 
> I'd expect that Cygwin must remove CRLF symbols from backtick results,

You expected wrong - by default Cygwin does the same as Linux, which
does NOT remove CR from command substitution (only LF).

> but remove only LF symbol. It gets different results on Windows and
> Linux. Result on Windows has additional CR symbol and for example
> string comparision, which is working in Linux, fails in Windows.

It is NOT a bug in Cygwin, but in your failure to sanitize your input.
You'd get the same behavior on Linux if you pass in CR bytes.  That said,

> 
> Is any kind of an option/setting which removes such CRs or it is a bug
> in Cygwin?

Read the release announcements: Cygwin bash DOES have an 'igncr' shell
option (set -o igncr) that does exactly what you are asking for - it
makes bash ignore CR characters in command substitutions and other places:

https://cygwin.com/ml/cygwin-announce/2017-01/msg00047.html

The option is not on by default, but exists because it is a common
desire when you can't be bothered to otherwise sanitize CR out of your
input stream.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Extra CR symbol from backticks on Cygwin 2.9.0

[Nikolay Melekhin]

Hi Eric,

Thanks! This option is what I am looking for.
However I cannot agree with you on bash behavior in Cygwin. Section of
command substitution in bash documentation says:

https://www.gnu.org/software/bash/manual/bash.html#Command-Substitution

"...with any trailing newlines deleted." That is why I expected that
bash should remove trailing newlines in command substitution results,
which are CRLF on Windows.
From my point of view, bash in Cygwin should follow the same rule on
Linux and Windows: remove trailing newlines in command substitution.
So remove trailing CRLF on Windows.


Best regards,
Nikolay Melekhin

2017-09-12 1:00 GMT+03:00 Eric Blake <eblake@redhat.com>:
> On 09/11/2017 04:27 PM, Nikolay Melekhin wrote:
>>
>> I'd expect that Cygwin must remove CRLF symbols from backtick results,
>
> You expected wrong - by default Cygwin does the same as Linux, which
> does NOT remove CR from command substitution (only LF).
>
>> but remove only LF symbol. It gets different results on Windows and
>> Linux. Result on Windows has additional CR symbol and for example
>> string comparision, which is working in Linux, fails in Windows.
>
> It is NOT a bug in Cygwin, but in your failure to sanitize your input.
> You'd get the same behavior on Linux if you pass in CR bytes.  That said,
>
>>
>> Is any kind of an option/setting which removes such CRs or it is a bug
>> in Cygwin?
>
> Read the release announcements: Cygwin bash DOES have an 'igncr' shell
> option (set -o igncr) that does exactly what you are asking for - it
> makes bash ignore CR characters in command substitutions and other places:
>
> https://cygwin.com/ml/cygwin-announce/2017-01/msg00047.html
>
> The option is not on by default, but exists because it is a common
> desire when you can't be bothered to otherwise sanitize CR out of your
> input stream.
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3266
> Virtualization:  qemu.org | libvirt.org
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Extra CR symbol from backticks on Cygwin 2.9.0

[Andrey Repin]

Greetings, Nikolay Melekhin!

Please don't top-post. Thank you.


> Thanks! This option is what I am looking for.
> However I cannot agree with you on bash behavior in Cygwin. Section of
> command substitution in bash documentation says:

> https://www.gnu.org/software/bash/manual/bash.html#Command-Substitution

> "...with any trailing newlines deleted." That is why I expected that
> bash should remove trailing newlines in command substitution results,
> which are CRLF on Windows.
> From my point of view, bash in Cygwin should follow the same rule on
> Linux and Windows: remove trailing newlines in command substitution.
> So remove trailing CRLF on Windows.

Cygwin adhere to POSIX standard. Not Windows.


-- 
With best regards,
Andrey Repin
Tuesday, September 12, 2017 12:06:51

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Extra CR symbol from backticks on Cygwin 2.9.0

[cyg Simple]

On 9/12/2017 1:38 AM, Nikolay Melekhin wrote:
> Hi Eric,
> 
> Thanks! This option is what I am looking for.
> However I cannot agree with you on bash behavior in Cygwin. Section of
> command substitution in bash documentation says:
> 
> https://www.gnu.org/software/bash/manual/bash.html#Command-Substitution
> 
> "...with any trailing newlines deleted." That is why I expected that

In POSIX and newline is simply a LF.  The fact that your input from a
Windows cmd.exe contains CR has no bearing on POSIX.

> bash should remove trailing newlines in command substitution results,
> which are CRLF on Windows.

If bash were to follow Windows semantics instead of POSIX then your
point would be valid.  However Cygwin is a POSIX emulation so your point
in this case is false.

> From my point of view, bash in Cygwin should follow the same rule on
> Linux and Windows: remove trailing newlines in command substitution.
> So remove trailing CRLF on Windows.

Since Linux and Cygwin are both POSIX then Cygwin is doing what Linux
does.  You just need to adjust the differences by properly filtering
your input.

-- 
cyg Simple

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Extra CR symbol from backticks on Cygwin 2.9.0

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 848 bytes --]

On 09/12/2017 12:38 AM, Nikolay Melekhin wrote:
> https://www.gnu.org/software/bash/manual/bash.html#Command-Substitution
> 
> "...with any trailing newlines deleted." That is why I expected that
> bash should remove trailing newlines in command substitution results,
> which are CRLF on Windows.

No, newlines are LF.  CR are carriage returns, not newlines.

>From my point of view, bash in Cygwin should follow the same rule on
> Linux and Windows: remove trailing newlines in command substitution.

That's exactly what it does, newlines only by default.

> So remove trailing CRLF on Windows.

CR are not newlines.  Cygwin apps don't generate CR by default, so bash
doesn't strip CR by default.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Extra CR symbol from backticks on Cygwin 2.9.0

[cyg Simple]

On 9/12/2017 9:06 AM, Eric Blake wrote:
> On 09/12/2017 12:38 AM, Nikolay Melekhin wrote:
>> https://www.gnu.org/software/bash/manual/bash.html#Command-Substitution
>>
>> "...with any trailing newlines deleted." That is why I expected that
>> bash should remove trailing newlines in command substitution results,
>> which are CRLF on Windows.
> 
> No, newlines are LF.  CR are carriage returns, not newlines.
> 

Unfortunately the Google searches don't agree.  I started to make the
same exclaim but decided against it after doing a Google search for newline.

-- 
cyg Simple

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

RE: Extra CR symbol from backticks on Cygwin 2.9.0

[Michel LaBarre]

Nikolay, there is no ASCII newline/EOL character.  A "newline/EOL" is a context-dependent notion.

The fundamental CYGWIN position (Eric, Simple, Andrey): POSIX uses LF as EOL so POSIX-compliant behaviour in CYGWIN is to remove LF - windows be damned.

A slightly more generous interpretation might be:
  POSIX removes the "EOL", which in the context of pure-POSIX is LF but in the broader context of CYGWIN on Windows, EOL can be LF or CR-LF, so CYGWIN might consider removing either.
It would, however, break compliance with POSIX since a pure-POSIX program that happens to deal with CR in its data could be broken when running on CYGWIN.

Strict POSIX compliance REQUIRES CYGWIN to strip ONLY LF unless explicitly directed to break compliance by "igncr" - caveat emptor.

Not all situations of catering to Windows norms constitute endorsing programmer laziness however.  Sometimes, there is no elegant or efficient alternative. 

I integrated and supplemented an enterprise backup software product in 150 Unix(Solaris, SYSVR4) and Windows sites with the same scripts. The product's utilities have the same names whether in Unix or Windows but the inputs and outputs adhere to the respective environments e.g. EOL varies based on the OS.  Managing 50,000 lines of scripts (ksh, awk, perl,...) in a strictly Unix-compliant manner would require wrappers for all utilities both obscuring the code and/or adding substantial overhead to convert (sometimes massive) data at the shell level rather than at the OS runtime library call interface invoked within the product utilities.  Fortunately the tools I was using (MKS) transparently handled the mapping and use of either EOL.  So would have "igncr" had I been using CYGWIN.

(Besides EOL handling, another pervasive area was the recognition of executable files - in Unix by permissions and embedded codes vs name suffixes and associations in Windows (FTYPE/ASSOC/PATHEXT). MKS's recognition of Windows norms substantially avoided the messy workarounds;  CYGWIN would not have been so accommodating.  There were other areas.)

 CYGWIN's fundamental purpose is to serve POSIX compliant programming under Windows.  Any deviation from this has to be via explicit directives e.g. igncr, lastpipe, noacl, etc..  

Not trying to sound like a jerk, but I am still not clear as to why CYGWIN users are not using Linux unless they have to support code running in both POSIX and Windows environments in which case, the CYGWIN mission could be elaborated to mention facilitating portability to, and integration with, Windows which go beyond just standards compliance.  This might elevate deviations, such as "igncr", from being perceived as catering to the lazy, non-purist, unwashed masses rather than as genuinely valuable and essential elements of CYGWIN.

Strict POSIX compliance suits developers of self-contained vertical applications with minimal need for deviations; the whole application is safely ensconced within a POSIX cocoon.  On the other hand, developers integrating Windows applications and services over which they have no control may need more flexibility.

That being said, it has been generous on the part of CYGWIN implementors to recognise the CYGWIN audience for whom strict POSIX compliance is secondary and the main objective is to have useful tools under Windows that also support portability outside Windows.  Thank you.

Michel LaBarre 

  

> --
> cyg Simple
> 
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Extra CR symbol from backticks on Cygwin 2.9.0

[cyg Simple]

On 9/12/2017 8:11 PM, Michel LaBarre wrote:
> 
> Not trying to sound like a jerk, but I am still not clear as to why CYGWIN users are not using Linux unless they have to support code running in both POSIX and Windows environments in which case, the CYGWIN mission could be elaborated to mention facilitating portability to, and integration with, Windows which go beyond just standards compliance.  This might elevate deviations, such as "igncr", from being perceived as catering to the lazy, non-purist, unwashed masses rather than as genuinely valuable and essential elements of CYGWIN.
> 

Because there are vendors who supply applications that our employers
purchase and tell us to support it.  Those applications could be on
Linux or on Windows or whatever OS.  Having the same scripts to support
many various operations be exactly the same for each operation is
helpful from a maintenance POV.  If it works on Cygwin I can know that
it will work on Linux.  If it works on Linux it may or may not work in
Cygwin just because of the extra CR Windows is famous for.  If it works
on Linux it may or may not work on some other *nix OS but if that *nix
is POSIX compliant most likely it will especially if extensions weren't
used in the scripts.

> Strict POSIX compliance suits developers of self-contained vertical applications with minimal need for deviations; the whole application is safely ensconced within a POSIX cocoon.  On the other hand, developers integrating Windows applications and services over which they have no control may need more flexibility.
> 

Most have issues when they try to use Cygwin outside of the Cygwin
shell.  While Cygwin tries to be helpful with that method it isn't the
suggested method of use and has lack of testers for changes.  If you use
Cygwin outside of the Cygwin "ensconced POSIX cocoon" then when a test
version of Cygwin is released and a call for testers then you'd be
better served by testing and reporting issues than being surprised when
updating after it is released.

> That being said, it has been generous on the part of CYGWIN implementors to recognise the CYGWIN audience for whom strict POSIX compliance is secondary and the main objective is to have useful tools under Windows that also support portability outside Windows.  Thank you.
> 

Cygwin has never been totally empathetic to Windows executables.  There
are many things that work but for each one that works there is another
that won't.  You can't expect that a Windows executable to understand
the POSIX PATH emulation for instance.  If you try to mix and match
executables between Cygwin and Windows you may have luck with a
particular version but later find that it no longer works because some
small change now causes you issues.  Live inside the Cygwin environment
as much as possible and limit the amount of pure Windows applications
you use.  I know there are many times when it's preferable to use a
Windows version versus a Cygwin version of an application gvIm is one I
use as a Windows app but I create a script to manage the PATH given to
the gvim.exe application.  When playing with Windows applications you
have to be willing to work around the differences, it is usually
possible and if you have issues with trying to do so then this list is
here to help.

-- 
cyg Simple

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Extra CR symbol from backticks on Cygwin 2.9.0

[Brian Inglis]

On 2017-09-13 08:34, cyg Simple wrote:
> On 9/12/2017 8:11 PM, Michel LaBarre wrote:
>> Not trying to sound like a jerk, but I am still not clear as to why CYGWIN 
>> users are not using Linux unless they have to support code running in both
>> POSIX and Windows environments in which case, the CYGWIN mission could be
>> elaborated to mention facilitating portability to, and integration with,
>> Windows which go beyond just standards compliance. This might elevate
>> deviations, such as "igncr", from being perceived as catering to the lazy,
>> non-purist, unwashed masses rather than as genuinely valuable and essential
>> elements of CYGWIN.
> Because there are vendors who supply applications that our employers
> purchase and tell us to support it.  Those applications could be on
> Linux or on Windows or whatever OS.  Having the same scripts to support
> many various operations be exactly the same for each operation is
> helpful from a maintenance POV.  If it works on Cygwin I can know that
> it will work on Linux.  If it works on Linux it may or may not work in
> Cygwin just because of the extra CR Windows is famous for.  If it works
> on Linux it may or may not work on some other *nix OS but if that *nix
> is POSIX compliant most likely it will especially if extensions weren't
> used in the scripts.
>> Strict POSIX compliance suits developers of self-contained vertical 
>> applications with minimal need for deviations; the whole application is
>> safely ensconced within a POSIX cocoon. On the other hand, developers
>> integrating Windows applications and services over which they have no
>> control may need more flexibility.
> Most have issues when they try to use Cygwin outside of the Cygwin
> shell.  While Cygwin tries to be helpful with that method it isn't the
> suggested method of use and has lack of testers for changes.  If you use
> Cygwin outside of the Cygwin "ensconced POSIX cocoon" then when a test
> version of Cygwin is released and a call for testers then you'd be
> better served by testing and reporting issues than being surprised when
> updating after it is released.
>> That being said, it has been generous on the part of CYGWIN implementors 
>> to recognise the CYGWIN audience for whom strict POSIX compliance is
>> secondary and the main objective is to have useful tools under Windows that
>> also support portability outside Windows. Thank you.
> Cygwin has never been totally empathetic to Windows executables.  There
> are many things that work but for each one that works there is another
> that won't.  You can't expect that a Windows executable to understand
> the POSIX PATH emulation for instance.  If you try to mix and match
> executables between Cygwin and Windows you may have luck with a
> particular version but later find that it no longer works because some
> small change now causes you issues.  Live inside the Cygwin environment
> as much as possible and limit the amount of pure Windows applications
> you use.  I know there are many times when it's preferable to use a
> Windows version versus a Cygwin version of an application gvIm is one I
> use as a Windows app but I create a script to manage the PATH given to
> the gvim.exe application.  When playing with Windows applications you
> have to be willing to work around the differences, it is usually
> possible and if you have issues with trying to do so then this list is
> here to help.

+1 for Windows native gvim with wrappers, and other Windows native GUIs like
BitKeeper, Tortoise git and Hg, Firefox, Thunderbird, Libre/OpenOffice, etc.
many of which can be invoked on files with cygstart due to auto-path-conversion.

I have also found Windows Subsystem for Linux/Bash for Windows useful mainly for
quick, convenient compatibility cross checks of scripts, source programs, and
packages.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple