RE: New sed in latest

RE: New sed in latest

[Heribert Dahms]

Hi Chris,

the misunderstanding seems to be on my side,
since od always dumping is what I expect, but
that isn't the same as the mentioned change to bash
with regard to smart ignoring of DOS line endings?

Bye, Heribert (heribert_dahms@icon-gmbh.de)

> -----Original Message-----
> From:	Chris Faylor [SMTP:cgf@cygnus.com]
> Sent:	Friday, June 02, 2000 23:13
> To:	'cygwin@sourceware.cygnus.com'
> Subject:	Re: New sed in latest
> 
> On Fri, Jun 02, 2000 at 10:51:55PM +0200, Heribert Dahms wrote:
> >what about something like CYGWIN=force_binary for those who need it?
> >If you change od, I'd like that and/or --binary, since I routinely
> use
> >od -c to check for \r.
> 
> I don't understand the question.  I'm talking about *fixing* tools
> like 'od'
> which do not always open their contents in binary mode as they should.
> 
> I'm not going to be sweeping through all of the tools provided with
> cygwin
> and indiscriminately making them read their input in text mode.
> 
> od was probably a bad example since I don't know if it is broken or
> not.
> The last time I checked, I thought it was, but I have just checked it
> and
> it is working as expected.
> 
> So 'od' does not have to be touched.  It correctly opens the file in
> binary
> mode and accurately dumps the contents.
> 
> cgf
> 
> --
> Want to unsubscribe from this list?
> Send a message to cygwin-unsubscribe@sourceware.cygnus.com

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Chris Faylor]

On Sun, Jun 04, 2000 at 05:06:42PM -0400, Christopher Jones wrote:
>This really seems broken if the way I mount something affects line endings
>such that I can't remove \r on a text mounted system with sed.  I can't
>think of a place where I really want to use cygwin to do DOS things... but
>if I did wouldn't I have to handle \r\n on my own just as I would if I were
>to manipulate a dos partition from GNU/Linux?  At least on UNIX how the file
>gets written doesn't change line endings on the way to disk.  Yuck.

You could *never* remove \r line endings on a text mounted system with sed
or any other program which does not go out of its way to open a file in
"binary mode".

cgf

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

RE: New sed in latest

[Christopher Jones]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1262 bytes --]

Title: RE: New sed in latest







> -----Original Message-----
> From: Corinna Vinschen [ mailto:corinna@vinschen.de ]
> Sent: Saturday, June 03, 2000 4:47 AM
> To: Bob McGowan
> Cc: cygwin@sourceware.cygnus.com
> Subject: Re: New sed in latest
> 
> 
> Bob McGowan wrote:
> > But sed writes to standard output.  So the correct answer 
> (I think) is
> > that, to capture the "dos2unix" output, the script would need to be
> > redirected to a file.  If the file is in a binary mounted 
> environment,
> > then it will be UNIX format, if it is text mounted it would 
> continue to
> > be DOS format.
> > 
> > Is this an accurate analysis?
> 
> Indeed. To get rid of the \r you only have to start sed now,
> doing nothing but writing it's input to a binary mounted
> output file.
> 
> Corinna


This really seems broken if the way I mount something affects line endings such that I can't remove \r on a text mounted system with sed.  I can't think of a place where I really want to use cygwin to do DOS things... but if I did wouldn't I have to handle \r\n on my own just as I would if I were to manipulate a dos partition from GNU/Linux?  At least on UNIX how the file gets written doesn't change line endings on the way to disk.  Yuck.

Brian

Re: New sed in latest

[Corinna Vinschen]

Bob McGowan wrote:
> But sed writes to standard output.  So the correct answer (I think) is
> that, to capture the "dos2unix" output, the script would need to be
> redirected to a file.  If the file is in a binary mounted environment,
> then it will be UNIX format, if it is text mounted it would continue to
> be DOS format.
> 
> Is this an accurate analysis?

Indeed. To get rid of the \r you only have to start sed now,
doing nothing but writing it's input to a binary mounted
output file.

Corinna

-- 
Corinna Vinschen
Cygwin Developer
Cygnus Solutions, a Red Hat company

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Bob McGowan]

Earnie Boyd wrote:
> 
> --- Charles Wilson <cwilson@ece.gatech.edu> wrote:
> > So, with the new sed behavior the following is no longer going to work,
> > right?
> 
> No, not right.

It _looks_ like Charles is right....

> >
> > ---dos2unix----
> > #!/bin/bash
> > sed 's/^M//' $1
> > ---------------
> >
> > Where the '^M' is generated by pressing CTRL-V, CTRL-M.
> >
> 
> In text mode, it is the \n that terminates the read.  If there is a \r
> preceding the \n it will also be removed and the count read reduced by one.  If
> there is no preceding \r then the count returned is the number actually read.
> 

....because, on a cursory review, the sed script mentioned, which used
to find \r in the DOS format file and remove them, thus making a UNIX
format file, will no longer find them, since as you say they have been
"removed and the count reduced by one."

But sed writes to standard output.  So the correct answer (I think) is
that, to capture the "dos2unix" output, the script would need to be
redirected to a file.  If the file is in a binary mounted environment,
then it will be UNIX format, if it is text mounted it would continue to
be DOS format.

Is this an accurate analysis?

Thanks,

-- 
Bob McGowan
Staff Software Quality Engineer
VERITAS Software
rmcgowan@veritas.com

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Chris Faylor]

On Fri, Jun 02, 2000 at 10:51:55PM +0200, Heribert Dahms wrote:
>what about something like CYGWIN=force_binary for those who need it?
>If you change od, I'd like that and/or --binary, since I routinely use
>od -c to check for \r.

I don't understand the question.  I'm talking about *fixing* tools like 'od'
which do not always open their contents in binary mode as they should.

I'm not going to be sweeping through all of the tools provided with cygwin
and indiscriminately making them read their input in text mode.

od was probably a bad example since I don't know if it is broken or not.
The last time I checked, I thought it was, but I have just checked it and
it is working as expected.

So 'od' does not have to be touched.  It correctly opens the file in binary
mode and accurately dumps the contents.

cgf

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

RE: New sed in latest

[Heribert Dahms]

Hi Chris,

what about something like CYGWIN=force_binary for those who need it?
If you change od, I'd like that and/or --binary, since I routinely use
od -c to check for \r.
Do you have another suggestion how to find out (from e.g. bash) actual
line endings
present in input/output files or pipes for not (yet) text/binary smart
programs?

Bye, Heribert (heribert_dahms@icon-gmbh.de)

> -----Original Message-----
> From:	Chris Faylor [SMTP:cgf@cygnus.com]
> Sent:	Friday, June 02, 2000 20:24
> To:	cygwin@sourceware.cygnus.com
> Subject:	Re: New sed in latest
> 
> On Fri, Jun 02, 2000 at 08:11:41PM +0200, Pierre.Humblet@eurecom.fr
> wrote:
> >On Thu, 1 Jun 2000 17:48:39 -0400, Chris Faylor <cgf@cygnus.com>
> wrote:
> >>You have my guarantee that when the cygwin mailing list is filled
> with
> >>the plaintive cries of people who are trying to process their binary
> >><snip>
> >
> >The current plaintive voices are about a textmode processing bug in
> the
> >dll.  Masking that bug by making applications like sed ignore \r even
> >on binary mounted systems is only a reasonable TEMPORARY (band aid)
> >measure.  Until now users mounting binary file systems had the
> >assurance that the tools would behave as on Unix.  That was great, I
> >hope it will come back.  Otherwise do you plan to keep on "fixing"
> more
> >applications?  awk (for one) is also "broken" by \r\n :
> >
> >bash-2.04$ awk 'NF > 1' /u/hello.txt bash-2.04$ mount -b -f u:/ /u
> >bash-2.04$ awk 'NF > 1' /u/hello.txt hello bash-2.04$
> >
> >(hello.txt contains "hello \r\n")
> 
> Yes.  I plan on fixing more applications.  In fact, awk is next on my
> list.
> 
> This change was only partially motivated by the recent broken DLL.  We
> have always had many many many complaints about "sed not working".
> The
> 1.1.1 DLL just aggravates the potential problems with binmode versus
> textmode processing.  The problem won't go away when the DLL is fixed.
> 
> You may have noticed that we've recently made similar modifications to
> ash and bash.  I also expect that we will be making modifications to
> applications like 'od' which do not set binmode appropriately.
> 
> Of course, the wonderful thing about all of this is that this is free
> software.  You don't have to use it.  Anyone can build their own
> version
> of sed which, theoretically, is able to process a binary file.  It's
> not
> that hard at all.
> 
> cgf
> 
> --
> Want to unsubscribe from this list?
> Send a message to cygwin-unsubscribe@sourceware.cygnus.com

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Earnie Boyd]

--- Charles Wilson <cwilson@ece.gatech.edu> wrote:
> So, with the new sed behavior the following is no longer going to work,
> right?

No, not right.

> 
> ---dos2unix----
> #!/bin/bash
> sed 's/^M//' $1
> ---------------
> 
> Where the '^M' is generated by pressing CTRL-V, CTRL-M.
> 

In text mode, it is the \n that terminates the read.  If there is a \r
preceding the \n it will also be removed and the count read reduced by one.  If
there is no preceding \r then the count returned is the number actually read.

Cheers,

=====
---
   Earnie Boyd: < mailto:earnie_boyd@yahoo.com >
            __Cygwin: POSIX on Windows__
Cygwin Newbies: < http://www.freeyellow.com/members5/gw32/index.html >
           __Minimalist GNU for Windows__
  Mingw32 List: < http://www.egroups.com/group/mingw32/ >
    Mingw Home: < http://www.mingw.org/ >

__________________________________________________
Do You Yahoo!?
Yahoo! Photos -- now, 100 FREE prints!
http://photos.yahoo.com

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Chris Faylor]

On Fri, Jun 02, 2000 at 04:06:15PM -0400, Charles Wilson wrote:
>So, with the new sed behavior the following is no longer going to work,
>right?
>
>---dos2unix----
>#!/bin/bash
>sed 's/^M//' $1
>---------------
>
>Where the '^M' is generated by pressing CTRL-V, CTRL-M.

	      setup sed

should download the newest version of sed to your machine.

cgf

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Charles Wilson]

So, with the new sed behavior the following is no longer going to work,
right?

---dos2unix----
#!/bin/bash
sed 's/^M//' $1
---------------

Where the '^M' is generated by pressing CTRL-V, CTRL-M.

--Chuck

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Chris Faylor]

On Fri, Jun 02, 2000 at 08:11:41PM +0200, Pierre.Humblet@eurecom.fr wrote:
>On Thu, 1 Jun 2000 17:48:39 -0400, Chris Faylor <cgf@cygnus.com> wrote:
>>You have my guarantee that when the cygwin mailing list is filled with
>>the plaintive cries of people who are trying to process their binary
>><snip>
>
>The current plaintive voices are about a textmode processing bug in the
>dll.  Masking that bug by making applications like sed ignore \r even
>on binary mounted systems is only a reasonable TEMPORARY (band aid)
>measure.  Until now users mounting binary file systems had the
>assurance that the tools would behave as on Unix.  That was great, I
>hope it will come back.  Otherwise do you plan to keep on "fixing" more
>applications?  awk (for one) is also "broken" by \r\n :
>
>bash-2.04$ awk 'NF > 1' /u/hello.txt bash-2.04$ mount -b -f u:/ /u
>bash-2.04$ awk 'NF > 1' /u/hello.txt hello bash-2.04$
>
>(hello.txt contains "hello \r\n")

Yes.  I plan on fixing more applications.  In fact, awk is next on my list.

This change was only partially motivated by the recent broken DLL.  We
have always had many many many complaints about "sed not working".  The
1.1.1 DLL just aggravates the potential problems with binmode versus
textmode processing.  The problem won't go away when the DLL is fixed.

You may have noticed that we've recently made similar modifications to
ash and bash.  I also expect that we will be making modifications to
applications like 'od' which do not set binmode appropriately.

Of course, the wonderful thing about all of this is that this is free
software.  You don't have to use it.  Anyone can build their own version
of sed which, theoretically, is able to process a binary file.  It's not
that hard at all.

cgf

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Pierre.Humblet]

On Thu, 1 Jun 2000 17:48:39 -0400, Chris Faylor <cgf@cygnus.com> wrote:
> You have my guarantee that when the cygwin mailing list is filled with
> the plaintive cries of people who are trying to process their binary <snip>
  
The current plaintive voices are about a textmode processing bug in the dll.
Masking that bug by making applications like sed ignore \r even on binary
mounted systems is only a reasonable TEMPORARY (band aid) measure.
Until now users mounting binary file systems had the assurance that the
tools would behave as on Unix. That was great, I hope it will come back.
Otherwise do you plan to keep on "fixing" more applications? 
awk (for one) is also "broken" by \r\n :

bash-2.04$ awk 'NF > 1' /u/hello.txt
bash-2.04$ mount -b -f u:/ /u
bash-2.04$ awk 'NF > 1' /u/hello.txt
hello
bash-2.04$

(hello.txt contains "hello \r\n")

Pierre

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Corinna Vinschen]

Peter Ring wrote:
> I wouldn't mind much if sed silently just dropped the '\r' at the end of
> a line. But I would not like it to introduce '\r' in the output, and I
> most certainly wouldn't like it to chop a file at the first ^Z.

The output files are written according to the mount mode.

Corinna

-- 
Corinna Vinschen
Cygwin Developer
Cygnus Solutions, a Red Hat company

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

RE: New sed in latest

[Peter Ring]

The 'text' files on my WindowsNT box tend to have a '\n', and maybe also
a '\r' before that, as end-of-line character(s). For manual editing of
'text' files, we use emacs or TextPad, both of which work just fine with
Unix conventions about end-of-line.

Most of what I do with sed either assumes '\n' to be end-of-line, or
works whether or not a '\r' sits at the end of the line.

I wouldn't mind much if sed silently just dropped the '\r' at the end of
a line. But I would not like it to introduce '\r' in the output, and I
most certainly wouldn't like it to chop a file at the first ^Z.

kind regards
Peter Ring
Forlaget MAGNUS A/S
A Wolters Kluwer Company


-----Original Message-----
From: Corinna Vinschen [ mailto:corinna@vinschen.de ]
Sent: Thursday, June 01, 2000 11:21 PM
To: Pierre.Humblet@eurecom.fr
Cc: cygwin@sourceware.cygnus.com
Subject: Re: New sed in latest


<snip>

I don't think so. sed is a typical text tool and nearly
everyone expects sed working correct on text files and
you know what text files are under Windows.

> Also, does your change apply to piped stdin?

Yep.

> I would withdraw this comment if there was anything in the sed
> documentation to the effect that \r\n is equivalent to \n.

How shall this be? sed isn't developed for OSes with sick (IMHO)
difference between binary and text mode.

<snip>

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Chris Faylor]

On Thu, Jun 01, 2000 at 05:07:32PM -0500, Larry Hall (RFK Partners, Inc) wrote:
>At 03:52 PM 6/1/00, Pierre.Humblet@eurecom.fr wrote:
>>In http://sourceware.cygnus.com/ml/cygwin/2000-05/msg01160.html
>> >Corinna Vinschen wrote:
>>>I have uploaded a new version of sed to pub/cygwin/latest/sed on
>>>sourceware.cygnus.com.  <snip>
>>>
>>>The new feature of that version is that it reads all files always in
>>>text mode so that files are correctly read when they have dos file
>>>endings in binary mounted directories as well.
>>
>>Thanks for the effort Corinna.  Although I understand the reason isn't
>>this pushing it too far?  When working on a binary mounted system I
>>would expect sed to work exactly as on Unix.  Although perhaps rare,
>>aren't there legitimate reasons to want to keep a \r at the end of a
>>line?  Sed is often used in fairly complicated fashions that may now be
>>broken.  Also, does your change apply to piped stdin?  I would withdraw
>>this comment if there was anything in the sed documentation to the
>>effect that \r\n is equivalent to \n.
>
>Another point is that binary files with CTRL-Z will also stop
>"processing" prematurely as a result.  I suppose this becomes the
>argument for having a "binary" (or conversely a "text") flag for sed
>(and some other tools)...

You have my guarantee that when the cygwin mailing list is filled with
the plaintive cries of people who are trying to process their binary
files but are finding that all of the \r\n's in the files are being
converted to \n and the processing stops at the first ^Z, we will
take corrective steps to deal with this.

Until then, making sed operate correctly on text files is the greatest
good for the greatest number of people.

cgf


--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Corinna Vinschen]

Pierre.Humblet@eurecom.fr wrote:
> Thanks for the effort Corinna. Although I understand the
> reason isn't this pushing it too far? When working on a binary
> mounted system I would expect sed to work exactly as on Unix.
> Although perhaps rare, aren't there legitimate reasons to want
> to keep a \r at the end of a line? Sed is often used in fairly
> complicated fashions that may now be broken.

I don't think so. sed is a typical text tool and nearly
everyone expects sed working correct on text files and
you know what text files are under Windows.

> Also, does your change apply to piped stdin?

Yep.

> I would withdraw this comment if there was anything in the sed
> documentation to the effect that \r\n is equivalent to \n.

How shall this be? sed isn't developed for OSes with sick (IMHO)
difference between binary and text mode.

Corinna

-- 
Corinna Vinschen
Cygwin Developer
Cygnus Solutions, a Red Hat company

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Larry Hall (RFK Partners, Inc)]

At 03:52 PM 6/1/00, Pierre.Humblet@eurecom.fr wrote:
>In http://sourceware.cygnus.com/ml/cygwin/2000-05/msg01160.html
> >Corinna Vinschen wrote:
> >
> >I have uploaded a new version of sed to pub/cygwin/latest/sed
> >on sourceware.cygnus.com. <snip>
> >
> >The new feature of that version is that it reads
> >all files always in text mode so that files are
> >correctly read when they have dos file endings
> >in binary mounted directories as well.
>
>Thanks for the effort Corinna. Although I understand the 
>reason isn't this pushing it too far? When working on a binary
>mounted system I would expect sed to work exactly as on Unix.
>Although perhaps rare, aren't there legitimate reasons to want
>to keep a \r at the end of a line? Sed is often used in fairly
>complicated fashions that may now be broken.
>Also, does your change apply to piped stdin?
>I would withdraw this comment if there was anything in the sed 
>documentation to the effect that \r\n is equivalent to \n.
>
>
>Pierre



Another point is that binary files with CTRL-Z will also stop "processing"
prematurely as a result.  I suppose this becomes the argument for having
a "binary" (or conversely a "text") flag for sed (and some other tools)...



Larry Hall                              lhall@rfk.com
RFK Partners, Inc.                      http://www.rfk.com
118 Washington Street                   (508) 893-9779 - RFK Office
Holliston, MA 01746                     (508) 893-9889 - FAX
                                        (508) 560-1285 - cell phone



--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Chris Faylor]

On Thu, Jun 01, 2000 at 10:52:17PM +0200, Pierre.Humblet@eurecom.fr wrote:
>In http://sourceware.cygnus.com/ml/cygwin/2000-05/msg01160.html
>>Corinna Vinschen wrote:
>>
>>I have uploaded a new version of sed to pub/cygwin/latest/sed
>>on sourceware.cygnus.com. <snip>
>>
>>The new feature of that version is that it reads all files always in
>>text mode so that files are correctly read when they have dos file
>>endings in binary mounted directories as well.
>
>Thanks for the effort Corinna.  Although I understand the reason isn't
>this pushing it too far?  When working on a binary mounted system I
>would expect sed to work exactly as on Unix.  Although perhaps rare,
>aren't there legitimate reasons to want to keep a \r at the end of a
>line?  Sed is often used in fairly complicated fashions that may now be
>broken.  Also, does your change apply to piped stdin?  I would withdraw
>this comment if there was anything in the sed documentation to the
>effect that \r\n is equivalent to \n.

Corinna did this at my request.  The bottom line is that this will solve
thousands of problems while possibly breaking a handful of uses.

If it becomes an issue we can always add a --binary option to sed.

cgf

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com

Re: New sed in latest

[Pierre.Humblet]

In http://sourceware.cygnus.com/ml/cygwin/2000-05/msg01160.html
>Corinna Vinschen wrote:
>
>I have uploaded a new version of sed to pub/cygwin/latest/sed
>on sourceware.cygnus.com. <snip>
>
>The new feature of that version is that it reads
>all files always in text mode so that files are
>correctly read when they have dos file endings
>in binary mounted directories as well.

Thanks for the effort Corinna. Although I understand the 
reason isn't this pushing it too far? When working on a binary
mounted system I would expect sed to work exactly as on Unix.
Although perhaps rare, aren't there legitimate reasons to want
to keep a \r at the end of a line? Sed is often used in fairly
complicated fashions that may now be broken.
Also, does your change apply to piped stdin?
I would withdraw this comment if there was anything in the sed 
documentation to the effect that \r\n is equivalent to \n.


Pierre

--
Want to unsubscribe from this list?
Send a message to cygwin-unsubscribe@sourceware.cygnus.com