cpp w/DOS line feeds

cpp w/DOS line feeds

[Matthew Kirkwood]

Hi,

Is there a reason that cpp doesn't belive MSDOS line breaks?

I ask because twice this week I have had to 
	tr -d '\r'
files which were last edited in a DOS editor of some sort.

Almost everything works, except for multi-line #defines:

#define MULTI_LINE_THING(a,b)	( (a) > (b) ? \^M
				  (a) : (b))

cpp thinks that the \ is escaping the ^M (\r in polite
company).

Is there a reason or standard which mandates this?

Matthew.

Re: cpp w/DOS line feeds

[Matthias Urlichs]

Hi,

Matthew Kirkwood:
> Is there a reason that cpp doesn't belive MSDOS line breaks?
> 
Probably "nobody thought about fixing it yet".

This is relative to egcs-1.1.1; somebody else will have to think about cpplib.

Index: base.210/gcc/cccp.c
--- base.210/gcc/cccp.c Mon, 10 Aug 1998 08:35:35 +0200 smurf (gnu_egcs/v/b/20_cccp.c 1.22.1.1 666)
+++ noris.14/gcc/cccp.c Wed, 11 Nov 1998 02:53:56 +0100 smurf (gnu_egcs/v/b/20_cccp.c 1.22.1.3 660)
@@ -1053,7 +1053,8 @@
    retrying if necessary.  If MAX_READ_LEN is defined, read at most
    that bytes at a time.  Return a negative value if an error occurs,
    otherwise return the actual number of bytes read,
-   which must be LEN unless end-of-file was reached.  */
+   which may be < LEN if CRs have been skipped, though we try not to do
+   that.  */
 
 static int
 safe_read (desc, ptr, len)
@@ -1062,6 +1063,7 @@
      int len;
 {
   int left, rcount, nchars;
+  char *rptr;
 
   left = len;
   while (left > 0) {
@@ -1081,8 +1083,20 @@
       }
     if (nchars == 0)
       break;
-    ptr += nchars;
+
+    /* CRLF pairs, found with Unix when processing DOS files,
+       throw off backslash-newline removal.
+       Therefore, CRs are thrown away here. */
     left -= nchars;
+    rptr = ptr;
+    while(nchars--)
+      {
+	if(*rptr == '\r')
+	  left++;
+	else
+	  *ptr++ = *rptr;
+	rptr++;
+      }
   }
   return len - left;
 }
@@ -2197,8 +2211,8 @@
     for (;;) {
       cnt = safe_read (f, (char *) fp->buf + size, bsize - size);
       if (cnt < 0) goto perror;	/* error! */
+      if (cnt == 0) break; /* End of file */
       size += cnt;
-      if (size != bsize) break;	/* End of file */
       bsize *= 2;
       fp->buf = (U_CHAR *) xrealloc (fp->buf, bsize + 2);
     }
@@ -4988,6 +5002,8 @@
 	  map_list_ptr->map_list_map = ptr;
 
 	  while ((ch = getc (f)) != '\n')
+	    if (ch == '\r')
+		continue;
 	    if (ch == EOF)
 	      break;
 	}
@@ -5220,9 +5236,9 @@
       i = safe_read (f, (char *) fp->buf + st_size, bsize - st_size);
       if (i < 0)
 	goto nope;      /* error! */
-      st_size += i;
-      if (st_size != bsize)
+      if (i == 0)
 	break;	/* End of file */
+      st_size += i;
       bsize *= 2;
       fp->buf = (U_CHAR *) xrealloc (fp->buf, bsize + 2);
     }

-- 
Matthias Urlichs  |  noris network GmbH   |   smurf@noris.de  |  ICQ: 20193661
The quote was selected randomly. Really.    |      http://www.noris.de/~smurf/
-- 
No one remembers learning how to use a spoon, it is something that is
learned and not taught.

Re: cpp w/DOS line feeds

[Matthew Kirkwood]

On Thu, 21 Jan 1999, Matthias Urlichs wrote:

> > Is there a reason that cpp doesn't belive MSDOS line breaks?
>
> Probably "nobody thought about fixing it yet".
>
> This is relative to egcs-1.1.1; somebody else will have to think about cpplib.
[snip]

Great stuff, thanks.

Matthew.

Re: cpp w/DOS line feeds

[Per Bothner]

How does your patch handle lone CR?  This should be handled as
a line-terminator as well.  (Lone CR is used a line-terminator
on MacOS, plus a few other rare/obsolete OSes.)  Therfore, just
throwing away CR does not do the Right Thing.

	--Per Bothner
Cygnus Solutions     bothner@cygnus.com     http://www.cygnus.com/~bothner

Re: cpp w/DOS line feeds

[Joe Buck]

> How does your patch handle lone CR?  This should be handled as
> a line-terminator as well.  (Lone CR is used a line-terminator
> on MacOS, plus a few other rare/obsolete OSes.)  Therfore, just
> throwing away CR does not do the Right Thing.

Unix has \n, DOS/Windows has \r\n, Mac has \r.  All three should be
treated as if \n were typed.

Re: cpp w/DOS line feeds

[Zack Weinberg]

On Thu, 21 Jan 99 10:09:06 PST, Joe Buck wrote:
>
>> How does your patch handle lone CR?  This should be handled as
>> a line-terminator as well.  (Lone CR is used a line-terminator
>> on MacOS, plus a few other rare/obsolete OSes.)  Therfore, just
>> throwing away CR does not do the Right Thing.
>
>Unix has \n, DOS/Windows has \r\n, Mac has \r.  All three should be
>treated as if \n were typed.

I have experimental code for cpplib that makes it understand all three
possible line endings (and, coincidentally, trigraphs).  I'm planning to
whip it into shape within the next few days.

zw

Re: cpp w/DOS line feeds

[Matthias Urlichs]

Hi,

Per Bothner:
> How does your patch handle lone CR?

It doesn't, which is one reason I didn't bother to send it in before.

(It should also handle \n\r, which is what you get when you read DOS files
on a Mac.  :-/  )

The other reason is that I think the standalone cpp is destined to be 
obsolete anyway.

The third reason is, of course, that there are no more Macintoshes around
here...

-- 
Matthias Urlichs  |  noris network GmbH   |   smurf@noris.de  |  ICQ: 20193661
The quote was selected randomly. Really.    |      http://www.noris.de/~smurf/
-- 
"Most of the dogmatic religions have exhibited a perverse talent for taking
 the wrong side on the most important concepts in the material universe,
 from the structure of the solar system to the origin of man."
		-- George Gaylord Simpson

Re: cpp w/DOS line feeds

[Matthias Urlichs]

Hi,

Zack Weinberg:
> I have experimental code for cpplib that makes it understand all three
> possible line endings

Four. When you compile and/or run the code on a Mac, you might get the
equivalent of \n\r when reading DOS files.

> (and, coincidentally, trigraphs).  I'm planning to
> whip it into shape within the next few days.
> 
That'd be cool.

-- 
Matthias Urlichs  |  noris network GmbH   |   smurf@noris.de  |  ICQ: 20193661
The quote was selected randomly. Really.    |      http://www.noris.de/~smurf/
-- 
The apathy of the born freeman is worse
than the docility of the born slave.
                        -- Grant Singleton (1890-?)

Re: cpp w/DOS line feeds

[Per Bothner]

> Four. When you compile and/or run the code on a Mac, you might get the
> equivalent of \n\r when reading DOS files.

Presumably only when opening the file in text mode - not when opining
it in binary mode?

	--Per Bothner
Cygnus Solutions     bothner@cygnus.com     http://www.cygnus.com/~bothner

Re: cpp w/DOS line feeds

[Matthias Urlichs]

Hi,

Per Bothner:
> > Four. When you compile and/or run the code on a Mac, you might get the
> > equivalent of \n\r when reading DOS files.
> 
> Presumably only when opening the file in text mode - not when opining
> it in binary mode?
> 
I've heard of at least one compiler which switches the ASCII codes
of \r and \n.  Thus, if you use code compiled by this thing, and if you
open the file in text mode, it would come out OK again.  :-/

-- 
Matthias Urlichs  |  noris network GmbH   |   smurf@noris.de  |  ICQ: 20193661
The quote was selected randomly. Really.    |      http://www.noris.de/~smurf/
-- 
Everyone complains of his memory, no one of his judgement.

Re: cpp w/DOS line feeds

[Matthias Urlichs]

Hi,

Matthew Kirkwood:
> Is there a reason that cpp doesn't belive MSDOS line breaks?
> 
Probably "nobody thought about fixing it yet".

This is relative to egcs-1.1.1; somebody else will have to think about cpplib.

Index: base.210/gcc/cccp.c
--- base.210/gcc/cccp.c Mon, 10 Aug 1998 08:35:35 +0200 smurf (gnu_egcs/v/b/20_cccp.c 1.22.1.1 666)
+++ noris.14/gcc/cccp.c Wed, 11 Nov 1998 02:53:56 +0100 smurf (gnu_egcs/v/b/20_cccp.c 1.22.1.3 660)
@@ -1053,7 +1053,8 @@
    retrying if necessary.  If MAX_READ_LEN is defined, read at most
    that bytes at a time.  Return a negative value if an error occurs,
    otherwise return the actual number of bytes read,
-   which must be LEN unless end-of-file was reached.  */
+   which may be < LEN if CRs have been skipped, though we try not to do
+   that.  */
 
 static int
 safe_read (desc, ptr, len)
@@ -1062,6 +1063,7 @@
      int len;
 {
   int left, rcount, nchars;
+  char *rptr;
 
   left = len;
   while (left > 0) {
@@ -1081,8 +1083,20 @@
       }
     if (nchars == 0)
       break;
-    ptr += nchars;
+
+    /* CRLF pairs, found with Unix when processing DOS files,
+       throw off backslash-newline removal.
+       Therefore, CRs are thrown away here. */
     left -= nchars;
+    rptr = ptr;
+    while(nchars--)
+      {
+	if(*rptr == '\r')
+	  left++;
+	else
+	  *ptr++ = *rptr;
+	rptr++;
+      }
   }
   return len - left;
 }
@@ -2197,8 +2211,8 @@
     for (;;) {
       cnt = safe_read (f, (char *) fp->buf + size, bsize - size);
       if (cnt < 0) goto perror;	/* error! */
+      if (cnt == 0) break; /* End of file */
       size += cnt;
-      if (size != bsize) break;	/* End of file */
       bsize *= 2;
       fp->buf = (U_CHAR *) xrealloc (fp->buf, bsize + 2);
     }
@@ -4988,6 +5002,8 @@
 	  map_list_ptr->map_list_map = ptr;
 
 	  while ((ch = getc (f)) != '\n')
+	    if (ch == '\r')
+		continue;
 	    if (ch == EOF)
 	      break;
 	}
@@ -5220,9 +5236,9 @@
       i = safe_read (f, (char *) fp->buf + st_size, bsize - st_size);
       if (i < 0)
 	goto nope;      /* error! */
-      st_size += i;
-      if (st_size != bsize)
+      if (i == 0)
 	break;	/* End of file */
+      st_size += i;
       bsize *= 2;
       fp->buf = (U_CHAR *) xrealloc (fp->buf, bsize + 2);
     }

-- 
Matthias Urlichs  |  noris network GmbH   |   smurf@noris.de  |  ICQ: 20193661
The quote was selected randomly. Really.    |      http://www.noris.de/~smurf/
-- 
No one remembers learning how to use a spoon, it is something that is
learned and not taught.

Re: cpp w/DOS line feeds

[Matthew Kirkwood]

On Thu, 21 Jan 1999, Matthias Urlichs wrote:

> > Is there a reason that cpp doesn't belive MSDOS line breaks?
>
> Probably "nobody thought about fixing it yet".
>
> This is relative to egcs-1.1.1; somebody else will have to think about cpplib.
[snip]

Great stuff, thanks.

Matthew.

Re: cpp w/DOS line feeds

[Matthias Urlichs]

Hi,

Zack Weinberg:
> I have experimental code for cpplib that makes it understand all three
> possible line endings

Four. When you compile and/or run the code on a Mac, you might get the
equivalent of \n\r when reading DOS files.

> (and, coincidentally, trigraphs).  I'm planning to
> whip it into shape within the next few days.
> 
That'd be cool.

-- 
Matthias Urlichs  |  noris network GmbH   |   smurf@noris.de  |  ICQ: 20193661
The quote was selected randomly. Really.    |      http://www.noris.de/~smurf/
-- 
The apathy of the born freeman is worse
than the docility of the born slave.
                        -- Grant Singleton (1890-?)

Re: cpp w/DOS line feeds

[Per Bothner]

How does your patch handle lone CR?  This should be handled as
a line-terminator as well.  (Lone CR is used a line-terminator
on MacOS, plus a few other rare/obsolete OSes.)  Therfore, just
throwing away CR does not do the Right Thing.

	--Per Bothner
Cygnus Solutions     bothner@cygnus.com     http://www.cygnus.com/~bothner

Re: cpp w/DOS line feeds

[Matthias Urlichs]

Hi,

Per Bothner:
> > Four. When you compile and/or run the code on a Mac, you might get the
> > equivalent of \n\r when reading DOS files.
> 
> Presumably only when opening the file in text mode - not when opining
> it in binary mode?
> 
I've heard of at least one compiler which switches the ASCII codes
of \r and \n.  Thus, if you use code compiled by this thing, and if you
open the file in text mode, it would come out OK again.  :-/

-- 
Matthias Urlichs  |  noris network GmbH   |   smurf@noris.de  |  ICQ: 20193661
The quote was selected randomly. Really.    |      http://www.noris.de/~smurf/
-- 
Everyone complains of his memory, no one of his judgement.

Re: cpp w/DOS line feeds

[Per Bothner]

> Four. When you compile and/or run the code on a Mac, you might get the
> equivalent of \n\r when reading DOS files.

Presumably only when opening the file in text mode - not when opining
it in binary mode?

	--Per Bothner
Cygnus Solutions     bothner@cygnus.com     http://www.cygnus.com/~bothner

Re: cpp w/DOS line feeds

[Zack Weinberg]

On Thu, 21 Jan 99 10:09:06 PST, Joe Buck wrote:
>
>> How does your patch handle lone CR?  This should be handled as
>> a line-terminator as well.  (Lone CR is used a line-terminator
>> on MacOS, plus a few other rare/obsolete OSes.)  Therfore, just
>> throwing away CR does not do the Right Thing.
>
>Unix has \n, DOS/Windows has \r\n, Mac has \r.  All three should be
>treated as if \n were typed.

I have experimental code for cpplib that makes it understand all three
possible line endings (and, coincidentally, trigraphs).  I'm planning to
whip it into shape within the next few days.

zw

Re: cpp w/DOS line feeds

[Joe Buck]

> How does your patch handle lone CR?  This should be handled as
> a line-terminator as well.  (Lone CR is used a line-terminator
> on MacOS, plus a few other rare/obsolete OSes.)  Therfore, just
> throwing away CR does not do the Right Thing.

Unix has \n, DOS/Windows has \r\n, Mac has \r.  All three should be
treated as if \n were typed.

Re: cpp w/DOS line feeds

[Matthias Urlichs]

Hi,

Per Bothner:
> How does your patch handle lone CR?

It doesn't, which is one reason I didn't bother to send it in before.

(It should also handle \n\r, which is what you get when you read DOS files
on a Mac.  :-/  )

The other reason is that I think the standalone cpp is destined to be 
obsolete anyway.

The third reason is, of course, that there are no more Macintoshes around
here...

-- 
Matthias Urlichs  |  noris network GmbH   |   smurf@noris.de  |  ICQ: 20193661
The quote was selected randomly. Really.    |      http://www.noris.de/~smurf/
-- 
"Most of the dogmatic religions have exhibited a perverse talent for taking
 the wrong side on the most important concepts in the material universe,
 from the structure of the solar system to the origin of man."
		-- George Gaylord Simpson

cpp w/DOS line feeds

[Matthew Kirkwood]

Hi,

Is there a reason that cpp doesn't belive MSDOS line breaks?

I ask because twice this week I have had to 
	tr -d '\r'
files which were last edited in a DOS editor of some sort.

Almost everything works, except for multi-line #defines:

#define MULTI_LINE_THING(a,b)	( (a) > (b) ? \^M
				  (a) : (b))

cpp thinks that the \ is escaping the ^M (\r in polite
company).

Is there a reason or standard which mandates this?

Matthew.

Re: cpp w/DOS line feeds

[Stuart Summerville (NEC)]

Hi all,

I'm currently contemplating the problem in gcc (or cpp in particular) of
multi-line macros that are not interpreted properly if the EOL is made up of
CRs as well as LFs. I've seen the informative thread from 21-Jan-99 about
this, with some bandied solutions, but nothing since then. Couldn't find
much in the gcc-bugs or gcc-announce lists either.

So, whats the current status on this issue? I'm currently running gcc
V2.91.57, & hope that 2.95.x contains the fix. Speaking of which, is the CPP
actually part of the gcc package? If not, where is it located? How does one
find out the version of their CPP tool?

Also, how does one search through web-based release notes between V2.91 and
V2.95 looking for such things?

Regards, sTu.
_________________________________________________________________________
Stuart Summerville                         NEC Australia Pty. Ltd.
ph: (+61 3) 9264-3090                      Integrated Comm Products (R&D)
fax:(+61 3) 9264-3841                      649-655 Springvale Road
e-mail: stuarts@icpdd.neca.nec.com.au      Mulgrave, VIC 3170, AUSTRALIA

Re: cpp w/DOS line feeds

[N8TM]

In a message dated 12/9/99 1:38:24 AM Pacific Standard Time, 
stuarts@icpdd.neca.nec.com.au writes:

> contemplating the problem in gcc (or cpp in particular) of
>  multi-line macros that are not interpreted properly if the EOL is made up 
of
>  CRs as well as LFs.
>hope that 2.95.x contains the fix

Don't hope too much.  As mentioned in the earlier discussion, cpp is intended 
to be able to use any character preceded by backslash to stand for that 
character taken literally, in the old unix tradition.

I just had an episode where source code containing \r compiled without 
complaints but didn't run identically when compiled under W2K and under 
linux.  Removing all the \r characters didn't change the results on Windows 
but corrected the variance in the linux version.  Maybe the 2.96 versions of 
the compiler are tolerant of \r when running on Windows but not otherwise.

The commercial compilers I have used work with source code without \r, but 
their run time may require text data files to contain \r.

Tim
tprince@computer.org

Re: cpp w/DOS line feeds

[Per Bothner]

N8TM@aol.com writes:

> In a message dated 12/9/99 1:38:24 AM Pacific Standard Time, 
> stuarts@icpdd.neca.nec.com.au writes:
> 
> > contemplating the problem in gcc (or cpp in particular) of
> >  multi-line macros that are not interpreted properly if the EOL is made up 
> of
> >  CRs as well as LFs.
> >hope that 2.95.x contains the fix
> 
> Don't hope too much.  As mentioned in the earlier discussion, cpp is intended 
> to be able to use any character preceded by backslash to stand for that 
> character taken literally, in the old unix tradition.

Except of course for newline.  And earlier discissions agreed that
CR-LF should be treated as equivalent to NL.  In fact, I believe
this is the currently implemented behavior if you enable cpplib
as a replacement for the old cpp.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: cpp w/DOS line feeds

[Dave Brolley]

Stuart Summerville (NEC) wrote:

> Hi all,
>
> I'm currently contemplating the problem in gcc (or cpp in particular) of
> multi-line macros that are not interpreted properly if the EOL is made up of
> CRs as well as LFs. I've seen the informative thread from 21-Jan-99 about
> this, with some bandied solutions, but nothing since then. Couldn't find
> much in the gcc-bugs or gcc-announce lists either.
>
> So, whats the current status on this issue? I'm currently running gcc
> V2.91.57, & hope that 2.95.x contains the fix.

Looks like this had been fixed in the cpplib version of the compilers
(configured with --enable-cpplib or --enable-c-cpplib) but not in cccp.c.

Dave

Re: cpp w/DOS line feeds

[Per Bothner]

N8TM@aol.com writes:

> In a message dated 12/9/99 1:38:24 AM Pacific Standard Time, 
> stuarts@icpdd.neca.nec.com.au writes:
> 
> > contemplating the problem in gcc (or cpp in particular) of
> >  multi-line macros that are not interpreted properly if the EOL is made up 
> of
> >  CRs as well as LFs.
> >hope that 2.95.x contains the fix
> 
> Don't hope too much.  As mentioned in the earlier discussion, cpp is intended 
> to be able to use any character preceded by backslash to stand for that 
> character taken literally, in the old unix tradition.

Except of course for newline.  And earlier discissions agreed that
CR-LF should be treated as equivalent to NL.  In fact, I believe
this is the currently implemented behavior if you enable cpplib
as a replacement for the old cpp.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/~per/

Re: cpp w/DOS line feeds

[Dave Brolley]

Stuart Summerville (NEC) wrote:

> Hi all,
>
> I'm currently contemplating the problem in gcc (or cpp in particular) of
> multi-line macros that are not interpreted properly if the EOL is made up of
> CRs as well as LFs. I've seen the informative thread from 21-Jan-99 about
> this, with some bandied solutions, but nothing since then. Couldn't find
> much in the gcc-bugs or gcc-announce lists either.
>
> So, whats the current status on this issue? I'm currently running gcc
> V2.91.57, & hope that 2.95.x contains the fix.

Looks like this had been fixed in the cpplib version of the compilers
(configured with --enable-cpplib or --enable-c-cpplib) but not in cccp.c.

Dave

Re: cpp w/DOS line feeds

[N8TM]

In a message dated 12/9/99 1:38:24 AM Pacific Standard Time, 
stuarts@icpdd.neca.nec.com.au writes:

> contemplating the problem in gcc (or cpp in particular) of
>  multi-line macros that are not interpreted properly if the EOL is made up 
of
>  CRs as well as LFs.
>hope that 2.95.x contains the fix

Don't hope too much.  As mentioned in the earlier discussion, cpp is intended 
to be able to use any character preceded by backslash to stand for that 
character taken literally, in the old unix tradition.

I just had an episode where source code containing \r compiled without 
complaints but didn't run identically when compiled under W2K and under 
linux.  Removing all the \r characters didn't change the results on Windows 
but corrected the variance in the linux version.  Maybe the 2.96 versions of 
the compiler are tolerant of \r when running on Windows but not otherwise.

The commercial compilers I have used work with source code without \r, but 
their run time may require text data files to contain \r.

Tim
tprince@computer.org

Re: cpp w/DOS line feeds

[Stuart Summerville (NEC)]

Hi all,

I'm currently contemplating the problem in gcc (or cpp in particular) of
multi-line macros that are not interpreted properly if the EOL is made up of
CRs as well as LFs. I've seen the informative thread from 21-Jan-99 about
this, with some bandied solutions, but nothing since then. Couldn't find
much in the gcc-bugs or gcc-announce lists either.

So, whats the current status on this issue? I'm currently running gcc
V2.91.57, & hope that 2.95.x contains the fix. Speaking of which, is the CPP
actually part of the gcc package? If not, where is it located? How does one
find out the version of their CPP tool?

Also, how does one search through web-based release notes between V2.91 and
V2.95 looking for such things?

Regards, sTu.
_________________________________________________________________________
Stuart Summerville                         NEC Australia Pty. Ltd.
ph: (+61 3) 9264-3090                      Integrated Comm Products (R&D)
fax:(+61 3) 9264-3841                      649-655 Springvale Road
e-mail: stuarts@icpdd.neca.nec.com.au      Mulgrave, VIC 3170, AUSTRALIA