sed match DOS end of line

sed match DOS end of line

[Paul Becker]

> From: On Behalf Of Corinna Vinschen
> Sent: Thursday, June 27, 2013 6:49 AM
> Subject: [ANNOUNCEMENT] Updated: sed-4.2.2-2
> 
> I've just updated the Cygwin 32 and 64 bit version of sed to 4.2.2-2.

Since this 'sed' change, I noticed that "$" does not match a dos formatted file end-of-line anymore.

example:
echo "endofline^M" | sed "s/e$/X/"

sed 4.2.1 returns
   endoflinX

sed 4.2.2 returns
   endofline

I don’t know if this is better or worse, or expected or not, but it is easy enough to fix my broken scripts.
Scripts will be less portable between windows and linux now unless I take this into consideration.
(and replace "$" with "^M*$")

$ echo $LC_COLLATE

$ echo $LANG
en_US.UTF-8

--
Paul

Re: sed match DOS end of line

[Corinna Vinschen]

On Jun 27 13:28, Paul Becker wrote:
> > From: On Behalf Of Corinna Vinschen
> > Sent: Thursday, June 27, 2013 6:49 AM
> > Subject: [ANNOUNCEMENT] Updated: sed-4.2.2-2
> > 
> > I've just updated the Cygwin 32 and 64 bit version of sed to 4.2.2-2.
> 
> Since this 'sed' change, I noticed that "$" does not match a dos formatted file end-of-line anymore.
> 
> example:
> echo "endofline^M" | sed "s/e$/X/"
> 
> sed 4.2.1 returns
>    endoflinX
> 
> sed 4.2.2 returns
>    endofline

This seems to be an upstream change.  The Cygwin 4.2.1 and 4.2.2
releases are build the same way.  4.2.1-2 (-3 on 64 bit) and 4.2.2-1 are
both built from just the vanilla upstream sources.

While 4.2.2-2 adds the latest upstream gnulib regex, it behaves
otherwise identically to 4.2.2-1.  And both 4.2.2 releases show the same
behaviour in terms of CRLF line endings.

But it's really puzzeling.  There's no indication for this change
in the sources.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

autoreconf/gettext puzzle (was Re: sed match DOS end of line)

[Corinna Vinschen]

On Jun 27 16:24, Corinna Vinschen wrote:
> On Jun 27 13:28, Paul Becker wrote:
> > > From: On Behalf Of Corinna Vinschen
> > > Sent: Thursday, June 27, 2013 6:49 AM
> > > Subject: [ANNOUNCEMENT] Updated: sed-4.2.2-2
> > > 
> > > I've just updated the Cygwin 32 and 64 bit version of sed to 4.2.2-2.
> > 
> > Since this 'sed' change, I noticed that "$" does not match a dos formatted file end-of-line anymore.
> > 
> > example:
> > echo "endofline^M" | sed "s/e$/X/"
> > 
> > sed 4.2.1 returns
> >    endoflinX
> > 
> > sed 4.2.2 returns
> >    endofline
> 
> This seems to be an upstream change.  The Cygwin 4.2.1 and 4.2.2
> releases are build the same way.  4.2.1-2 (-3 on 64 bit) and 4.2.2-1 are
> both built from just the vanilla upstream sources.
> 
> While 4.2.2-2 adds the latest upstream gnulib regex, it behaves
> otherwise identically to 4.2.2-1.  And both 4.2.2 releases show the same
> behaviour in terms of CRLF line endings.
> 
> But it's really puzzeling.  There's no indication for this change
> in the sources.

Ok, here's me, even more puzzled.  FIW, the 4.2.2 packages have *not*
been built the same way as the 4.2.1 package.  The difference is running
autoreconf (4.2.2) vs. not running autoreconf (4.2.1).

If I build sed with the auto configury unchanged, I'm back to an sed
which handles the CR just as the old sed 4.2.1 package:

  $ sed --version | head -1
  sed (GNU sed) 4.2.2
  $ od -c eol
  0000000   e   n   d   o   f   l   i   n   e  \r  \n
  0000013
  $ sed 's/e$/X/' < eol
  endoflinX

However, 4.2.2-1 and 4.2.2-2 packages have been built running autoreconf
first, which, besides other things, updates the included gettext stuff
from version 0.17 to version 0.18.1.

So I'm wondering if the latest gettext is the culprit here, but I'm not
fluent enough in the differences within gettext to see where the problem
is.

Chuck?  Yaakov?  Do you have any explanation or, even better, a
workaround for this phenomenon, other than not running autoreconf?

Anyway, for the time being, I'll upload 4.2.2-3 packages shortly which
has been build without running autoreconf, which will fix the CRLF vs.
LF problem.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: autoreconf/gettext puzzle (was Re: sed match DOS end of line)

[Corinna Vinschen]

On Jun 27 16:49, Corinna Vinschen wrote:
> On Jun 27 16:24, Corinna Vinschen wrote:
> > On Jun 27 13:28, Paul Becker wrote:
> > > > From: On Behalf Of Corinna Vinschen
> > > > Sent: Thursday, June 27, 2013 6:49 AM
> > > > Subject: [ANNOUNCEMENT] Updated: sed-4.2.2-2
> > > > 
> > > > I've just updated the Cygwin 32 and 64 bit version of sed to 4.2.2-2.
> > > 
> > > Since this 'sed' change, I noticed that "$" does not match a dos formatted file end-of-line anymore.
> > > 
> > > example:
> > > echo "endofline^M" | sed "s/e$/X/"
> > > 
> > > sed 4.2.1 returns
> > >    endoflinX
> > > 
> > > sed 4.2.2 returns
> > >    endofline
> > 
> > This seems to be an upstream change.  The Cygwin 4.2.1 and 4.2.2
> > releases are build the same way.  4.2.1-2 (-3 on 64 bit) and 4.2.2-1 are
> > both built from just the vanilla upstream sources.
> > 
> > While 4.2.2-2 adds the latest upstream gnulib regex, it behaves
> > otherwise identically to 4.2.2-1.  And both 4.2.2 releases show the same
> > behaviour in terms of CRLF line endings.
> > 
> > But it's really puzzeling.  There's no indication for this change
> > in the sources.
> 
> Ok, here's me, even more puzzled.  FIW, the 4.2.2 packages have *not*
> been built the same way as the 4.2.1 package.  The difference is running
> autoreconf (4.2.2) vs. not running autoreconf (4.2.1).
> 
> If I build sed with the auto configury unchanged, I'm back to an sed
> which handles the CR just as the old sed 4.2.1 package:
> 
>   $ sed --version | head -1
>   sed (GNU sed) 4.2.2
>   $ od -c eol
>   0000000   e   n   d   o   f   l   i   n   e  \r  \n
>   0000013
>   $ sed 's/e$/X/' < eol
>   endoflinX
> 
> However, 4.2.2-1 and 4.2.2-2 packages have been built running autoreconf
> first, which, besides other things, updates the included gettext stuff
> from version 0.17 to version 0.18.1.
> 
> So I'm wondering if the latest gettext is the culprit here, but I'm not
> fluent enough in the differences within gettext to see where the problem
> is.
> 
> Chuck?  Yaakov?  Do you have any explanation or, even better, a
> workaround for this phenomenon, other than not running autoreconf?
> 
> Anyway, for the time being, I'll upload 4.2.2-3 packages shortly which
> has been build without running autoreconf, which will fix the CRLF vs.
> LF problem.

Easier said than done.  This only fixed CRLF handling for the 64 bit
version.  The 32 bit version, built without autoreconf, still doesn't
recognize the CR.

Help?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: autoreconf/gettext puzzle (was Re: sed match DOS end of line)

[Yaakov (Cygwin/X)]

On 2013-06-27 09:49, Corinna Vinschen wrote:
> Ok, here's me, even more puzzled.  FIW, the 4.2.2 packages have *not*
> been built the same way as the 4.2.1 package.  The difference is running
> autoreconf (4.2.2) vs. not running autoreconf (4.2.1).

Were these natively compiled or cross-compiled?  The latter pulls in 
gnulib functions in place of libc's (Cygwin) without intervention during 
configure.

> If I build sed with the auto configury unchanged, I'm back to an sed
> which handles the CR just as the old sed 4.2.1 package:
>
>    $ sed --version | head -1
>    sed (GNU sed) 4.2.2
>    $ od -c eol
>    0000000   e   n   d   o   f   l   i   n   e  \r  \n
>    0000013
>    $ sed 's/e$/X/' < eol
>    endoflinX
>
> However, 4.2.2-1 and 4.2.2-2 packages have been built running autoreconf
> first, which, besides other things, updates the included gettext stuff
> from version 0.17 to version 0.18.1.
>
> So I'm wondering if the latest gettext is the culprit here, but I'm not
> fluent enough in the differences within gettext to see where the problem
> is.

I doubt it; as long as libintl is dynamically linked, the gettext build 
infrastructure just handles .po.mo rules.


Yaakov


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: autoreconf/gettext puzzle (was Re: sed match DOS end of line)

[Corinna Vinschen]

On Jun 27 10:42, Yaakov (Cygwin/X) wrote:
> On 2013-06-27 09:49, Corinna Vinschen wrote:
> >Ok, here's me, even more puzzled.  FIW, the 4.2.2 packages have *not*
> >been built the same way as the 4.2.1 package.  The difference is running
> >autoreconf (4.2.2) vs. not running autoreconf (4.2.1).
> 
> Were these natively compiled or cross-compiled?  The latter pulls in
> gnulib functions in place of libc's (Cygwin) without intervention
> during configure.
> 
> >If I build sed with the auto configury unchanged, I'm back to an sed
> >which handles the CR just as the old sed 4.2.1 package:
> >
> >   $ sed --version | head -1
> >   sed (GNU sed) 4.2.2
> >   $ od -c eol
> >   0000000   e   n   d   o   f   l   i   n   e  \r  \n
> >   0000013
> >   $ sed 's/e$/X/' < eol
> >   endoflinX
> >
> >However, 4.2.2-1 and 4.2.2-2 packages have been built running autoreconf
> >first, which, besides other things, updates the included gettext stuff
> >from version 0.17 to version 0.18.1.
> >
> >So I'm wondering if the latest gettext is the culprit here, but I'm not
> >fluent enough in the differences within gettext to see where the problem
> >is.
> 
> I doubt it; as long as libintl is dynamically linked, the gettext
> build infrastructure just handles .po.mo rules.

Yeah, I seem to be entirely on the wrong track, sorry about that.

I'm just inspecting the sources and it seems that the read mode of
files has changed from "rt" to "r" between 4.2.1 and 4.2.2 for some
reason.

I'm still digging, and I'm unsure how I could create a version of
sed 4.2.2 which did the right thing at all a couple of minutes ago.

Still digging...


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: autoreconf/gettext puzzle (was Re: sed match DOS end of line)

[Corinna Vinschen]

On Jun 27 17:51, Corinna Vinschen wrote:
> On Jun 27 10:42, Yaakov (Cygwin/X) wrote:
> > On 2013-06-27 09:49, Corinna Vinschen wrote:
> > >Ok, here's me, even more puzzled.  FIW, the 4.2.2 packages have *not*
> > >been built the same way as the 4.2.1 package.  The difference is running
> > >autoreconf (4.2.2) vs. not running autoreconf (4.2.1).
> > 
> > Were these natively compiled or cross-compiled?  The latter pulls in
> > gnulib functions in place of libc's (Cygwin) without intervention
> > during configure.
> > 
> > >If I build sed with the auto configury unchanged, I'm back to an sed
> > >which handles the CR just as the old sed 4.2.1 package:
> > >
> > >   $ sed --version | head -1
> > >   sed (GNU sed) 4.2.2
> > >   $ od -c eol
> > >   0000000   e   n   d   o   f   l   i   n   e  \r  \n
> > >   0000013
> > >   $ sed 's/e$/X/' < eol
> > >   endoflinX
> > >
> > >However, 4.2.2-1 and 4.2.2-2 packages have been built running autoreconf
> > >first, which, besides other things, updates the included gettext stuff
> > >from version 0.17 to version 0.18.1.
> > >
> > >So I'm wondering if the latest gettext is the culprit here, but I'm not
> > >fluent enough in the differences within gettext to see where the problem
> > >is.
> > 
> > I doubt it; as long as libintl is dynamically linked, the gettext
> > build infrastructure just handles .po.mo rules.
> 
> Yeah, I seem to be entirely on the wrong track, sorry about that.
> 
> I'm just inspecting the sources and it seems that the read mode of
> files has changed from "rt" to "r" between 4.2.1 and 4.2.2 for some
> reason.
> 
> I'm still digging, and I'm unsure how I could create a version of
> sed 4.2.2 which did the right thing at all a couple of minutes ago.
> 
> Still digging...

I have no idea how I screwed up so badly while testing.  It was actually
just a missing "t" flag in a call to fopen which has been lost between
4.2.1 and 4.2.2.

I just uploaded and announced 4.2.2-3 which should fix this issue, and I
sent the patch upstream.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: sed match DOS end of line

[Steven Penny]

On Sat, 6 Jan 2018 16:25:20, Steve Harlow wrote:
> Do I not have some environment variable set correctly? Is this a bug in sed?
> 
> I see that I can install the older version, 4.2.2.3.  Looks like that 
> might take a step backwards on some libraries too. compiler-rt(5.0.1-1), 
> libc++devel (5.0.1-1), libc++1 (5.0.1-1), etc.  Is that the right way to go?

nope - this is documented and has been discuessed at length already - past
behavior was wrong if you were on binary mount and/or pipe. current behavior is
correct for those cases:

http://cygwin.com/ml/cygwin/2017-06/msg00044.html


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

sed match DOS end of line

[Steve Harlow]

Greetings,

With 'sed' (GNU sed) version 4.4 I am seeing that '$' doesn't match end 
of line with windows type text files.
It looks like something like this happened and was fixed on version 4.2.2.
https://cygwin.com/ml/cygwin/2013-06/msg00685.html

Here is the original complaint at the time.  I'm seeing the same thing 
with version 4.4.
https://cygwin.com/ml/cygwin/2013-06/msg00673.html

Do I not have some environment variable set correctly? Is this a bug in sed?

I see that I can install the older version, 4.2.2.3.  Looks like that 
might take a step backwards on some libraries too. compiler-rt(5.0.1-1), 
libc++devel (5.0.1-1), libc++1 (5.0.1-1), etc.  Is that the right way to go?

Regards,
Steve


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple