gawk Regression: CR characters are not stripped on Windows

gawk Regression: CR characters are not stripped on Windows

[Orgad Shaneh]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

Hi,

Cross-posting per Eli Zaretskii's request.

CR characters used to be automatically stripped on Windows (MSYS2 and
Cygwin environments). This is broken in 4.2.0.

Minimal example:
echo -en "foo\r\n\r\nbar\r\n" > foo.txt
awk '/^$/ { print "found" }' foo.txt # This worked with 4.1.4 and
doesn't work with 4.2.0
awk '/^\r$/ { print "found" }' foo.txt # This works with 4.2.0 and
doesn't work with 4.1.4

Bisected to commit 5db38f775d9ba239e125d81dff2010a2ddacb48e:
(* gawkmisc.c (cygwin_premain0, cygwin_premain2): Remove.
No longer needed).

Apparently it's still needed...

This issue was reported in https://github.com/git-for-windows/git/issues/1524

Proposed patch is attached.

As Eli said, this change was deliberate. But this has several drawbacks.

1. The gawk info page states that:

> Under MS-Windows, 'gawk' (and many other text programs) silently
> translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on
> output.

and on Feb 8 the following section was added:

> Recent versions of Cygwin open all files in binary mode.  This means
> that you should use 'RS = "\r?\n"' in order to be able to handle
> standard MS-Windows text files with carriage-return plus line-feed line
> endings.

This breaks compatibility between different gawk versions. What were
the reasons for this change in cygwin, and why was it pushed upstream?

2. Git and other tools automatically convert text files to CRLF on
Windows. This means that any awk script that runs on both platforms
must use RS = "\r?\n". One example that was broken by this behavior
change is gerrit's commit-msg hook[1], which scans for empty lines by
/^$/ regexp.

Please consider reverting this change. Patch attached.

[1] https://gerrit.googlesource.com/gerrit/+/376a7bbb64f1b3f13c261f4efa0af0e8538cfe9b/resources/com/google/gerrit/server/tools/root/hooks/commit-msg#101

- Orgad

[-- Attachment #2: 0001-Revert-default-mode-on-Cygwin-from-binary-back-to-te.patch --]
[-- Type: application/octet-stream, Size: 4649 bytes --]

[-- Attachment #3: Type: text/plain, Size: 219 bytes --]


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gawk Regression: CR characters are not stripped on Windows

[Andrey Repin]

Greetings, Orgad Shaneh!

> 1. The gawk info page states that:

>> Under MS-Windows,
------------^^^^^^^
>> 'gawk' (and many other text programs) silently
>> translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on
>> output.

> and on Feb 8 the following section was added:

>> Recent versions of Cygwin open all files in binary mode.  This means
>> that you should use 'RS = "\r?\n"' in order to be able to handle
>> standard MS-Windows text files with carriage-return plus line-feed line
>> endings.

> This breaks compatibility between different gawk versions. What were
> the reasons for this change in cygwin, and why was it pushed upstream?

> 2. Git and other tools automatically convert text files to CRLF on
> Windows.
--^^^^^^^

Cygwin is not "Windows", it is "sort of Linux".
Besides, this kind silent mangling is dangerous to an unsuspecting user.

> This means that any awk script that runs on both platforms
> must use RS = "\r?\n". One example that was broken by this behavior
> change is gerrit's commit-msg hook[1], which scans for empty lines by
> /^$/ regexp.


-- 
With best regards,
Andrey Repin
Tuesday, February 27, 2018 12:44:21

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gawk Regression: CR characters are not stripped on Windows

[Orgad Shaneh]

On Tue, Feb 27, 2018 at 11:48 AM, Andrey Repin <anrdaemon@yandex.ru> wrote:
> Greetings, Orgad Shaneh!
>
>> 1. The gawk info page states that:
>
>>> Under MS-Windows,
> ------------^^^^^^^
>>> 'gawk' (and many other text programs) silently
>>> translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on
>>> output.
>
>> and on Feb 8 the following section was added:
>
>>> Recent versions of Cygwin open all files in binary mode.  This means
>>> that you should use 'RS = "\r?\n"' in order to be able to handle
>>> standard MS-Windows text files with carriage-return plus line-feed line
>>> endings.
>
>> This breaks compatibility between different gawk versions. What were
>> the reasons for this change in cygwin, and why was it pushed upstream?
>
>> 2. Git and other tools automatically convert text files to CRLF on
>> Windows.
> --^^^^^^^
>
> Cygwin is not "Windows", it is "sort of Linux".
> Besides, this kind silent mangling is dangerous to an unsuspecting user.

I see. This is however not true for MSYS2.

Then I guess we will just keep this as a patch for MSYS2, which is
already merged[1]?

[1] https://github.com/Alexpux/MSYS2-packages/commit/c81d882b9838f8245603c7a8d5f8845eeadd6c2a

- Orgad

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gawk Regression: CR characters are not stripped on Windows

[Steven Penny]

On Tue, 27 Feb 2018 12:13:37, Orgad Shaneh wrote:
> I see. This is however not true for MSYS2.
>
> Then I guess we will just keep this as a patch for MSYS2, which is
> already merged[1]?
>
> [1] https://github.com/Alexpux/MSYS2-packages/commit/c81d882b9838f8245603c7a8d5f8845eeadd6c2a

yes, it is sad that you tricked him into doing that. by doing that youve now
rebroke something else:

http://cygwin.com/ml/cygwin/2017-02/msg00155.html

and while your gripe can by fixed by simply doing "awk something | tr -d '\r'",
the Awk issue that I presented (and that was properly fixed) is now broken again
with Alexpux/MSYS2-packages and git-for-windows/git.

so because of your refusal to add an extra pipe on your command line
(or just do text mounts), portable Awk scripts are broken again on those
platforms.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gawk Regression: CR characters are not stripped on Windows

[Houder]

On Tue, 27 Feb 2018 09:22:18, Orgad Shaneh wrote:

> Cross-posting per Eli Zaretskii's request.
> 
> CR characters used to be automatically stripped on Windows (MSYS2 and
> Cygwin environments). This is broken in 4.2.0.

4.2.0 ???? (change was made in 4.1.4)

Something for you to study (while Corinna is still busy with
different things):

    https://cygwin.com/ml/cygwin/2017-02/msg00243.html
    ( [ANNOUNCEMENT] Updated: gawk-4.1.4-3 )

At the same time sed, grep ... were modified accordingly.

Henri


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gawk Regression: CR characters are not stripped on Windows

[Brian Inglis]

On 2018-02-27 00:22, Orgad Shaneh wrote:
> Cross-posting per Eli Zaretskii's request.
> CR characters used to be automatically stripped on Windows (MSYS2 and
> Cygwin environments). This is broken in 4.2.0.

Cygwin binary mounts treat files as on Unix.

You missed all the discussions in early 2017 about gawk, grep, sed EOL handling:

https://sourceware.org/ml/cygwin/2017-02/msg00152.html
https://sourceware.org/ml/cygwin/2017-02/msg00188.html
https://sourceware.org/ml/cygwin/2017-02/msg00189.html

following on from discussions about bash after ShellShock:

https://sourceware.org/ml/cygwin/2016-08/msg00097.html

> Minimal example:
> echo -en "foo\r\n\r\nbar\r\n" > foo.txt
> awk '/^$/ { print "found" }' foo.txt # This worked with 4.1.4 and
> doesn't work with 4.2.0
> awk '/^\r$/ { print "found" }' foo.txt # This works with 4.2.0 and
> doesn't work with 4.1.4
>> Under MS-Windows, 'gawk' (and many other text programs) silently
>> translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on
>> output.

Cygwin does not try to be an MS Windows environment.
Cygwin tries its best to be a POSIX/Unix/Linux environment.

> and on Feb 8 the following section was added:
>> Recent versions of Cygwin open all files in binary mode.  This means
>> that you should use 'RS = "\r?\n"' in order to be able to handle
>> standard MS-Windows text files with carriage-return plus line-feed line
>> endings.

Use DOS files from a Cygwin text mount which does the conversion.

> This breaks compatibility between different gawk versions. What were
> the reasons for this change in cygwin, and why was it pushed upstream?

Compatibility with POSIX/Unix/Linux systems, except on a text mount, to allow
scripts which deal with binary data or embedded \r to work correctly, and
require scripts which work correctly, on Windows or Unix text as the application
provides, prefers, or ignores, and under Unix/Cygwin/Msys/Mingw.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: gawk Regression: CR characters are not stripped on Windows

[Eric Blake]

[urrgh - Cygwin's list policy in supplying reply-to makes it difficult 
to reply-to-all]

On 02/27/2018 01:22 AM, Orgad Shaneh wrote:
> Hi,
> 
> Cross-posting per Eli Zaretskii's request.
> 
> CR characters used to be automatically stripped on Windows (MSYS2 and
> Cygwin environments). This is broken in 4.2.0.

You should not think of Cygwin as a Windows environment, but as a 
Linux-alike environment.  gawk on Linux does not automatically strip 
CRs, therefore gawk on Cygwin should not automatically strip CRs.

What MSYS2 does is different, and that environment is entitled to use 
patches to make interoperability with native windows program nicer, at 
the expense of being less like Linux.

Furthermore, the change in Cygwin predates the gawk 4.2.0 release, and 
was intentionally made in a coordinated release in Feb 2017 alongside 
sed and grep:

https://sourceware.org/ml/cygwin/2017-02/msg00152.html
https://sourceware.org/ml/cygwin/2017-02/msg00188.html
https://sourceware.org/ml/cygwin/2017-02/msg00189.html

following on from discussions about bash after ShellShock:

https://sourceware.org/ml/cygwin/2016-08/msg00097.html

Changing gawk back to automatically strip CRs on Cygwin would be a 
regression.

> As Eli said, this change was deliberate. But this has several drawbacks.
> 
> 1. The gawk info page states that:
> 
>> Under MS-Windows, 'gawk' (and many other text programs) silently
>> translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on
>> output.
> 
> and on Feb 8 the following section was added:
> 
>> Recent versions of Cygwin open all files in binary mode.  This means
>> that you should use 'RS = "\r?\n"' in order to be able to handle
>> standard MS-Windows text files with carriage-return plus line-feed line
>> endings.

Or mount your Windows text files under a text mount in Cygwin (so that 
such files already have \r stripped), or add steps to your pipelines to 
strip CR before handing the data to gawk.

> 
> This breaks compatibility between different gawk versions. What were
> the reasons for this change in cygwin, and why was it pushed upstream?

See the discussion in Feb 2017 for rationale, but the executive summary 
is that Cygwin attempts to emulate Linux, silent corruption of binary 
files was deemed worse than manually having to explicitly strip CR when 
dealing with Windows text output.

> 
> 2. Git and other tools automatically convert text files to CRLF on
> Windows.

Not Cygwin git.  The problems you are encountering are more likely to 
happen when you mix and match tools from disparate environments, rather 
than when you use all tools from the same source.

> This means that any awk script that runs on both platforms
> must use RS = "\r?\n".

or strip the CR in any other means.  But the same is true of any script 
that must run on both Windows and Linux.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug-gawk] gawk Regression: CR characters are not stripped on Windows

[arnold]

Is there a way to distinguish cygwin from msys at compile time?
I would not object to restoring the behavior for msys only.

Thanks,

Arnold

Orgad Shaneh <orgads@gmail.com> wrote:

> Hi,
>
> Cross-posting per Eli Zaretskii's request.
>
> CR characters used to be automatically stripped on Windows (MSYS2 and
> Cygwin environments). This is broken in 4.2.0.
>
> Minimal example:
> echo -en "foo\r\n\r\nbar\r\n" > foo.txt
> awk '/^$/ { print "found" }' foo.txt # This worked with 4.1.4 and
> doesn't work with 4.2.0
> awk '/^\r$/ { print "found" }' foo.txt # This works with 4.2.0 and
> doesn't work with 4.1.4
>
> Bisected to commit 5db38f775d9ba239e125d81dff2010a2ddacb48e:
> (* gawkmisc.c (cygwin_premain0, cygwin_premain2): Remove.
> No longer needed).
>
> Apparently it's still needed...
>
> This issue was reported in https://github.com/git-for-windows/git/issues/1524
>
> Proposed patch is attached.
>
> As Eli said, this change was deliberate. But this has several drawbacks.
>
> 1. The gawk info page states that:
>
> > Under MS-Windows, 'gawk' (and many other text programs) silently
> > translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on
> > output.
>
> and on Feb 8 the following section was added:
>
> > Recent versions of Cygwin open all files in binary mode.  This means
> > that you should use 'RS = "\r?\n"' in order to be able to handle
> > standard MS-Windows text files with carriage-return plus line-feed line
> > endings.
>
> This breaks compatibility between different gawk versions. What were
> the reasons for this change in cygwin, and why was it pushed upstream?
>
> 2. Git and other tools automatically convert text files to CRLF on
> Windows. This means that any awk script that runs on both platforms
> must use RS = "\r?\n". One example that was broken by this behavior
> change is gerrit's commit-msg hook[1], which scans for empty lines by
> /^$/ regexp.
>
> Please consider reverting this change. Patch attached.
>
> [1] https://gerrit.googlesource.com/gerrit/+/376a7bbb64f1b3f13c261f4efa0af0e8538cfe9b/resources/com/google/gerrit/server/tools/root/hooks/commit-msg#101
>
> - Orgad

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug-gawk] gawk Regression: CR characters are not stripped on Windows

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 377 bytes --]

Hi Arnold,

On Mar  5 06:36, arnold@skeeve.com wrote:
> Is there a way to distinguish cygwin from msys at compile time?
> I would not object to restoring the behavior for msys only.

__MSYS__ vs. __CYGWIN__


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [bug-gawk] gawk Regression: CR characters are not stripped on Windows

[arnold]

Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:

> Hi Arnold,
>
> On Mar  5 06:36, arnold@skeeve.com wrote:
> > Is there a way to distinguish cygwin from msys at compile time?
> > I would not object to restoring the behavior for msys only.
>
> __MSYS__ vs. __CYGWIN__
>
>
> Corinna
>
> -- 
> Corinna Vinschen                  Please, send mails regarding Cygwin to
> Cygwin Maintainer                 cygwin AT cygwin DOT com
> Red Hat

Excellent. I will probably do that, soon.

Thanks!

Arnold

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug-gawk] gawk Regression: CR characters are not stripped on Windows

[arnold]

Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:

> Hi Arnold,
>
> On Mar  5 06:36, arnold@skeeve.com wrote:
> > Is there a way to distinguish cygwin from msys at compile time?
> > I would not object to restoring the behavior for msys only.
>
> __MSYS__ vs. __CYGWIN__
>
>
> Corinna
>
> -- 
> Corinna Vinschen                  Please, send mails regarding Cygwin to
> Cygwin Maintainer                 cygwin AT cygwin DOT com
> Red Hat

Excellent. I will probably do that, soon.

Thanks!

Arnold

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug-gawk] gawk Regression: CR characters are not stripped on Windows

[Andrey Repin]

Greetings, arnold@skeeve.com!

> Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:

>> Hi Arnold,
>>
>> On Mar  5 06:36, arnold@skeeve.com wrote:
>> > Is there a way to distinguish cygwin from msys at compile time?
>> > I would not object to restoring the behavior for msys only.
>>
>> __MSYS__ vs. __CYGWIN__
>>
>>
>> Corinna
>>
>> -- 
>> Corinna Vinschen                  Please, send mails regarding Cygwin to
>> Cygwin Maintainer                 cygwin AT cygwin DOT com
>> Red Hat

> Excellent. I will probably do that, soon.

Keep in mind that you should avoid relying on __CYGWIN__ if possible.
Only if certain POSIX mandated functionality could not be implemented at all
using WinAPI you may go for this macro to work around compatibility issues.


-- 
With best regards,
Andrey Repin
Monday, March 5, 2018 23:56:14

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug-gawk] gawk Regression: CR characters are not stripped on Windows

[Vince Rice]

> On Mar 5, 2018, at 2:58 PM, Andrey Repin wrote:
> 
> Greetings, <redacted>
> 
>> Corinna Vinschen wrote:
>> 
>>> Hi Arnold,
>>> 
>>> On Mar  5 06:36, arnold@skeeve.com wrote:
>>>> Is there a way to distinguish cygwin from msys at compile time?
>>>> I would not object to restoring the behavior for msys only.
>>> 
>>> __MSYS__ vs. __CYGWIN__
>>> 
>>> 
>>> Corinna
>> 
>> Excellent. I will probably do that, soon.
> 
> Keep in mind that you should avoid relying on __CYGWIN__ if possible.
> Only if certain POSIX mandated functionality could not be implemented at all
> using WinAPI you may go for this macro to work around compatibility issues.

https://cygwin.com/acronyms/#PCYMTNQREAIYR (see <redacted> above, which
wasn’t redacted in your reply).
(And Arnold, you did the same thing with Corinna’s email address.)

Keep in mind that it was Corinna answering the question. Given the subject of this
discussion, and that Corinna gave the answer, he doesn’t really need to “avoid
relying on __CYGWIN__”.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: [bug-gawk] gawk Regression: CR characters are not stripped on Windows

[arnold]

Andrey Repin <anrdaemon@yandex.ru> wrote:

> Greetings, arnold@skeeve.com!
>
> > Corinna Vinschen <corinna-cygwin@cygwin.com> wrote:
>
> >> Hi Arnold,
> >>
> >> On Mar  5 06:36, arnold@skeeve.com wrote:
> >> > Is there a way to distinguish cygwin from msys at compile time?
> >> > I would not object to restoring the behavior for msys only.
> >>
> >> __MSYS__ vs. __CYGWIN__
> >>
> >>
> >> Corinna
>
> > Excellent. I will probably do that, soon.
>
> Keep in mind that you should avoid relying on __CYGWIN__ if possible.
> Only if certain POSIX mandated functionality could not be implemented at all
> using WinAPI you may go for this macro to work around compatibility issues.

I don't think it's a problem; I am testing for MSYS.  The change
is now in git in posix/gawkmisc.c.

Thanks,

Arnold

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple