Re: When is \n converted to \r\n? and why?

Re: When is \n converted to \r\n? and why?

[Christopher G. Faylor]

In article <Pine.GSO.3.96.981028153437.966D-100000.cygnus.gnu-win32@fungus>,
Urban Widmark <urban@svenskatest.se> wrote:
>(using cygwin32 19.3 & 19.1?)
>
>If I create a small example file:
>
>$ echo a > xx
>$ ls -l xx
>-rw-r--r--   1 544      everyone        3 Oct 28 15:43 xx
>
>it will contain "a\r\n" since I use the default (non-binary mounts)
>
>But if I do:
>$ echo a | wc -c
>      2
>
>I get only 2 chars ... ok, so the translation is done when writing to
>disk.

Depending on the shell you're using, pipes will default to
binary mode.  That's obviously the case here.

>Then something like this will fail:
>
>$ echo a | tr -d '\r' > yy
>$ ls -l yy
>-rw-r--r--   1 544      everyone        3 Oct 28 15:48 yy
>
>So how am I supposed to remove the \r from the echo output?  Well, I know
>non-cygwin ways to remove the \r, the issue is if this is a tr bug, a

If you think that tr should operate on its input and output in a
binary fashion then it is a tr bug for not setting its stdout to
binary.  Possibly, setting CYGWIN32=binmode will work.

>It's also strange that writing to a pipe is different from writing to
>disk, that seems very non-unix'ish. I think it would be better if
>conversion could be toggled by changing an environment variable, it should
>not depend on the destination of my output.

See above.  How this operates also depends on whether the shell you're
using sets pipes to binmode or not.
-- 
cgf@cygnus.com
http://www.cygnus.com/
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: When is \n converted to \r\n? and why?

[Christopher G. Faylor]

In article <19981030020029.5367.rocketmail.cygnus.gnu-win32@send102.yahoomail.com>,
Earnie Boyd <earnie_boyd@yahoo.com> wrote:
>---Urban Widmark <urban@svenskatest.se> wrote:
>>
>> (using cygwin32 19.3 & 19.1?)
>> 
>> If I create a small example file:
>> 
>> $ echo a > xx
>> $ ls -l xx
>> -rw-r--r--   1 544      everyone        3 Oct 28 15:43 xx
>> 
>> it will contain "a\r\n" since I use the default (non-binary mounts)
>> 
>> But if I do:
>> $ echo a | wc -c
>>       2
>> 
>> I get only 2 chars ... ok, so the translation is done when writing to
>> disk. Then something like this will fail:
>
>Actually, the translation is done at the time of read.  The \r is
>removed when processing in text mode and the read would return 2 even
>though 3 was actully read from disk.

No.  In this case the stdout of the echo command was a pipe.  The
shell in use has put the pipe into binmode which means that it will
not add a \r before the \n.  IMO, pipes should always be opened
in binmode.

>> $ echo a | tr -d '\r' > yy
>> $ ls -l yy
>> -rw-r--r--   1 544      everyone        3 Oct 28 15:48 yy
>
>Well, another improperly ported utility.  `tr' should open it's input
>and output in binary mode; obviously from this example it isn't.

Right.  It should be calling setmode(1, O_BINARY).  When tr is invoked
it inherits an fd.  It has no idea if that fd is associated with a file
in a directory that is mounted "binary" or not by this time.  The only
thing it can do is call setmode() on its stdout fd.

>>It's also strange that writing to a pipe is different from writing to
>>disk, that seems very non-unix'ish.  I think it would be better if
>>conversion could be toggled by changing an environment variable, it
>>should not depend on the destination of my output.
>
>Well, you are at the mercy of the filesystem and text mode processing
>will prefix a \r to every \n.  This is only a problem if the programs
>are not properly ported.

The "filesystem" has nothing to do with this.  The filesystem is under
NT control.  It is cygwin which is writing the \r stuff.  What gets
written depends on how the directory you are writing to is mounted,
on the setting of the CYGWIN binmode option, and on options to the
open and setmode functions.

>>An environment variable would be better than a registry entry, since
>>it's hard to write "non-conversion" shell scripts that can be run on
>>other machines if they depend on a registry entry ...  Also some
>>scripts may need the feature turned on, others off.
>
>Actually, I'd be in favor of not having either if that meant getting
>properly ported code.  I wonder if we could get this resolved prior to
>the b20 release?

If you're asking for something other than CYGWIN=binmode then I'm afraid
this won't be happening.
-- 
cgf@cygnus.com
http://www.cygnus.com/
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: When is \n converted to \r\n? and why?

[Earnie Boyd]

---Urban Widmark <urban@svenskatest.se> wrote:
>
> On Thu, 29 Oct 1998, Larry Hall wrote:
> 
> > 
> > Nope.  Its not a tr or any other kind of bug.  You said it
yourself above.
> > Translation is done when writing to (and reading from) the disk. 
How can
> > tr be expected to remove something that doesn't exist.  Remember,
text/
> > binary modes have to do with how things are written to files on
the disk,
> > not how they are handled in stdin, stdout, pipes, and whatever.
> > 
> 
> Which means writing the file to disk and then doing cat is not the
same as
> piping it directly. I think that is wrong.
> 

Yep!  Me too.

> Or is the \r removed when reading a \r\n file from disk with text
mode? 
> Is it always removed or does it depend on the program I use (eg cat
> doesn't remove it, tr/wc does ...).

In text mode processing the read function will remove the \r and
return subtract it from the count of bytes read.  If the file contains
only \n it will be read also, the only difference is that the \r is
removed and not counted as read and a C-z will cause a EOF.

Unless you've personally modified them cat, tr and wc use the default
processing mode which you have appropriately set to text mode.

> 
> The amount of problems this will cause seems endless if I have to
remember
> which programs convert and which doesn't. Assuming it matters for
the end
> result, which it did in this case, the windows program wouldn't work
if
> the configuration files we tried to generate contained \r. A bug in
that
> software? definitely! But the cygwin behaviour is still confusing.
> 

What _WINDOWS_ program doesn't like a \r\n?  Please inform us as this
is very unwindows like.

> 
> Btw, I don't have anything against \r\n, I just don't like when
programs
> convert things automagically when I haven't told them to do so ... I
guess
> it's a control thingy. :)

No what you mean.  But, if you understand what is happening you can
adjust for it.

==
-                  \\||//
-------------o0O0--Earnie--0O0o--------------
--          earnie_boyd@yahoo.com          --
-- http://www.freeyellow.com/members5/gw32 --
----------------ooo0O--O0ooo-----------------

PS: Newbie's, you should visit my page.
_________________________________________________________
DO YOU YAHOO!?
Get your free @yahoo.com address at http://mail.yahoo.com

-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: When is \n converted to \r\n? and why?

[Urban Widmark]

On Sat, 31 Oct 1998, Earnie Boyd wrote:

> > Which means writing the file to disk and then doing cat is not the
> same as
> > piping it directly. I think that is wrong.
> > 
> 
> Yep!  Me too.

So I wasn't completely off target then ...


> What _WINDOWS_ program doesn't like a \r\n?  Please inform us as this
> is very unwindows like.
> 

It's a plugin for MS IIS called ServletExec ( http://www.newatlanta.com/ ),
we wanted to automate the configuration of our servlets by using a nice
bash script and write all the parameters, classpath etc to the config
files directly instead of using the graphical interface. This fails if the
created files contain \r\n.

But except for the pipes I guess this was just another FAQ which deserved
a RTFM answer (which I didn't get, someone must be having too much time on
their hands :).

The b20 FAQ is much better on this topic, except it doesn't say what the
binmode setting does, and it isn't mentioned in the "How is the DOS/Unix
CR/LF thing handled?" entry.

"... mode to binary via mount point options or by setting the
CYGWIN32 environment variable."
   changed to
"... mode to binary via mount point options or by setting binmode in the
CYGWIN32 environment variable."
   or something?


/Urban

---
Urban Widmark                           urban@svenskatest.se
Svenska Test AB                         +46 90 71 71 23


-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: When is \n converted to \r\n? and why?

[Earnie Boyd]

---Larry Hall <lhall@rfk.com> wrote:
>
> At 04:18 PM 10/28/98 +0100, Urban Widmark wrote:
> >(using cygwin32 19.3 & 19.1?)
> >
> >If I create a small example file:
> >
> >$ echo a > xx
> >$ ls -l xx
> >-rw-r--r--   1 544      everyone        3 Oct 28 15:43 xx
> >
> >it will contain "a\r\n" since I use the default (non-binary mounts)
> >
> >But if I do:
> >$ echo a | wc -c
> >      2
> >
> >I get only 2 chars ... ok, so the translation is done when writing to
> >disk. Then something like this will fail:
> >
> >$ echo a | tr -d '\r' > yy
> >$ ls -l yy
> >-rw-r--r--   1 544      everyone        3 Oct 28 15:48 yy
> >
> >So how am I supposed to remove the \r from the echo output?  Well,
I know
> >non-cygwin ways to remove the \r, the issue is if this is a tr bug, a
> >cygwin conversion bug or ...
> 
> Nope.  Its not a tr or any other kind of bug.  You said it yourself
above.
> Translation is done when writing to (and reading from) the disk. 
How can
> tr be expected to remove something that doesn't exist.  Remember,
text/
> binary modes have to do with how things are written to files on the
disk,
> not how they are handled in stdin, stdout, pipes, and whatever.
> 

`tr' should setmode(outdesc, O_BINARY) to avoid rewriting the \r.

==
-                  \\||//
-------------o0O0--Earnie--0O0o--------------
--          earnie_boyd@yahoo.com          --
-- http://www.freeyellow.com/members5/gw32 --
----------------ooo0O--O0ooo-----------------

PS: Newbie's, you should visit my page.
_________________________________________________________
DO YOU YAHOO!?
Get your free @yahoo.com address at http://mail.yahoo.com

-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: When is \n converted to \r\n? and why?

[Earnie Boyd]

---Urban Widmark <urban@svenskatest.se> wrote:
>
> (using cygwin32 19.3 & 19.1?)
> 
> If I create a small example file:
> 
> $ echo a > xx
> $ ls -l xx
> -rw-r--r--   1 544      everyone        3 Oct 28 15:43 xx
> 
> it will contain "a\r\n" since I use the default (non-binary mounts)
> 
> But if I do:
> $ echo a | wc -c
>       2
> 
> I get only 2 chars ... ok, so the translation is done when writing to
> disk. Then something like this will fail:
> 

Actually, the translation is done at the time of read.  The \r is
removed when processing in text mode and the read would return 2 even
though 3 was actully read from disk.

> $ echo a | tr -d '\r' > yy
> $ ls -l yy
> -rw-r--r--   1 544      everyone        3 Oct 28 15:48 yy
> 

Well, another improperly ported utility.  `tr' should open it's input
and output in binary mode; obviously from this example it isn't.

> So how am I supposed to remove the \r from the echo output?  Well, I
know
> non-cygwin ways to remove the \r, the issue is if this is a tr bug, a
> cygwin conversion bug or ...
> 
> 
> It's also strange that writing to a pipe is different from writing to
> disk, that seems very non-unix'ish. I think it would be better if
> conversion could be toggled by changing an environment variable, it
should
> not depend on the destination of my output.
> 

Well, you are at the mercy of the filesystem and text mode processing
will prefix a \r to every \n.  This is only a problem if the programs
are not properly ported.

> An environment variable would be better than a registry entry, since
it's
> hard to write "non-conversion" shell scripts that can be run on other
> machines if they depend on a registry entry ... Also some scripts
may need
> the feature turned on, others off.
> 

Actually, I'd be in favor of not having either if that meant getting
properly ported code.  I wonder if we could get this resolved prior to
the b20 release?

==
-                  \\||//
-------------o0O0--Earnie--0O0o--------------
--          earnie_boyd@yahoo.com          --
-- http://www.freeyellow.com/members5/gw32 --
----------------ooo0O--O0ooo-----------------

PS: Newbie's, you should visit my page.
_________________________________________________________
DO YOU YAHOO!?
Get your free @yahoo.com address at http://mail.yahoo.com

-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: When is \n converted to \r\n? and why?

[Christopher Faylor]

On Thu, Oct 29, 1998 at 06:00:29PM -0800, Earnie Boyd wrote:
>>An environment variable would be better than a registry entry, since
>>it's hard to write "non-conversion" shell scripts that can be run on
>>other machines if they depend on a registry entry ...  Also some
>>scripts may need the feature turned on, others off.
>
>Actually, I'd be in favor of not having either if that meant getting
>properly ported code.  I wonder if we could get this resolved prior to
>the b20 release?

e.g., CYGWIN=binmode ?

cgf
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

When is \n converted to \r\n? and why?

[Urban Widmark]

(using cygwin32 19.3 & 19.1?)

If I create a small example file:

$ echo a > xx
$ ls -l xx
-rw-r--r--   1 544      everyone        3 Oct 28 15:43 xx

it will contain "a\r\n" since I use the default (non-binary mounts)

But if I do:
$ echo a | wc -c
      2

I get only 2 chars ... ok, so the translation is done when writing to
disk. Then something like this will fail:

$ echo a | tr -d '\r' > yy
$ ls -l yy
-rw-r--r--   1 544      everyone        3 Oct 28 15:48 yy

So how am I supposed to remove the \r from the echo output?  Well, I know
non-cygwin ways to remove the \r, the issue is if this is a tr bug, a
cygwin conversion bug or ...


It's also strange that writing to a pipe is different from writing to
disk, that seems very non-unix'ish. I think it would be better if
conversion could be toggled by changing an environment variable, it should
not depend on the destination of my output.

An environment variable would be better than a registry entry, since it's
hard to write "non-conversion" shell scripts that can be run on other
machines if they depend on a registry entry ... Also some scripts may need
the feature turned on, others off.

/Urban, who knows next to nothing about windows .dlls

---
Urban Widmark                           urban@svenskatest.se
Svenska Test AB                         +46 90 71 71 23

-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: When is \n converted to \r\n? and why?

[Larry Hall]

At 04:18 PM 10/28/98 +0100, Urban Widmark wrote:
>(using cygwin32 19.3 & 19.1?)
>
>If I create a small example file:
>
>$ echo a > xx
>$ ls -l xx
>-rw-r--r--   1 544      everyone        3 Oct 28 15:43 xx
>
>it will contain "a\r\n" since I use the default (non-binary mounts)
>
>But if I do:
>$ echo a | wc -c
>      2
>
>I get only 2 chars ... ok, so the translation is done when writing to
>disk. Then something like this will fail:
>
>$ echo a | tr -d '\r' > yy
>$ ls -l yy
>-rw-r--r--   1 544      everyone        3 Oct 28 15:48 yy
>
>So how am I supposed to remove the \r from the echo output?  Well, I know
>non-cygwin ways to remove the \r, the issue is if this is a tr bug, a
>cygwin conversion bug or ...

Nope.  Its not a tr or any other kind of bug.  You said it yourself above.
Translation is done when writing to (and reading from) the disk.  How can
tr be expected to remove something that doesn't exist.  Remember, text/
binary modes have to do with how things are written to files on the disk,
not how they are handled in stdin, stdout, pipes, and whatever.


Larry
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: When is \n converted to \r\n? and why?

[Urban Widmark]

On Thu, 29 Oct 1998, Larry Hall wrote:

> 
> Nope.  Its not a tr or any other kind of bug.  You said it yourself above.
> Translation is done when writing to (and reading from) the disk.  How can
> tr be expected to remove something that doesn't exist.  Remember, text/
> binary modes have to do with how things are written to files on the disk,
> not how they are handled in stdin, stdout, pipes, and whatever.
> 

Which means writing the file to disk and then doing cat is not the same as
piping it directly. I think that is wrong.

Or is the \r removed when reading a \r\n file from disk with text mode? 
Is it always removed or does it depend on the program I use (eg cat
doesn't remove it, tr/wc does ...).

The amount of problems this will cause seems endless if I have to remember
which programs convert and which doesn't. Assuming it matters for the end
result, which it did in this case, the windows program wouldn't work if
the configuration files we tried to generate contained \r. A bug in that
software? definitely! But the cygwin behaviour is still confusing.


Btw, I don't have anything against \r\n, I just don't like when programs
convert things automagically when I haven't told them to do so ... I guess
it's a control thingy. :)

/Urban, unable to understand people who doesn't turn off autocorrection in
        word

---
Urban Widmark                           urban@svenskatest.se
Svenska Test AB                         +46 90 71 71 23

-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".