short fread(), but no ferror/feof

short fread(), but no ferror/feof

[Peter Åstrand]

I've discovered that fread(ptr, size, nitems, stream) sometimes returns a
value less than "nitems", but does not set feof() nor ferror(). As I
understand, this is incorrect: All fread() documentation I've found
confirms this. For example, IEEE Std 1003.1 says:

"Upon successful completion, fread() shall return the number of elements
successfully read which is less than nitems only if a read error or
end-of-file is encountered."

I've only noticed the problem when using fread() on a pipe, and when
setting a (another) file descriptor in the same application in
close-on-exec mode. The details are on http://python.org/sf/1071516. I've
been using the latest Cygwin (as of 2004-12-04) on Windows 98.

To me, this looks like a Cygwin bug. Any ideas?

/Peter Åstrand <astrand@lysator.liu.se>


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Christopher Faylor]

On Sun, Dec 05, 2004 at 05:17:46PM +0100, Peter ?strand wrote:
>I've discovered that fread(ptr, size, nitems, stream) sometimes returns a
>value less than "nitems", but does not set feof() nor ferror(). As I
>understand, this is incorrect: All fread() documentation I've found
>confirms this. For example, IEEE Std 1003.1 says:
>
>"Upon successful completion, fread() shall return the number of elements
>successfully read which is less than nitems only if a read error or
>end-of-file is encountered."
>
>I've only noticed the problem when using fread() on a pipe, and when
>setting a (another) file descriptor in the same application in
>close-on-exec mode. The details are on http://python.org/sf/1071516. I've
>been using the latest Cygwin (as of 2004-12-04) on Windows 98.
>
>To me, this looks like a Cygwin bug. Any ideas?

A simple, compilable test case with results shown on both linux and
cygwin would prove your theory.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Peter Astrand]

>> I've discovered that fread(ptr, size, nitems, stream) sometimes returns a
>> value less than "nitems", but does not set feof() nor ferror(). As I

>A simple, compilable test case with results shown on both linux and
>cygwin would prove your theory.

Here you go; example follows. Test run on Linux:

---
Got 66 bytes:
/bin/ls: /somebogus: Filen eller katalogen finns inte
/etc/passwd
|EOF has been reached.
---

Test run on Cygwin:

---
$ ./a.exe
Got 9 bytes:
/bin/ls: |
---

As you can see, the result is partial, and there is no EOF nor error.

I guess this is the problem described at
http://sources.redhat.com/ml/newlib/2004/msg00477.html. Perhaps you can
apply the suggested patch?

Example code follows.


#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <fcntl.h>


void checked_close(int fd)
{
    if (close(fd) == -1) {
	perror("close");
	exit(1);
    }
}


void checked_pipe(int filedes[2])
{
    if (pipe(filedes) == -1) {
	perror("filedes");
	exit(1);
    }
}


int checked_dup2(int oldfd, int newfd)
{
    int ret;
    if ((ret = dup2(oldfd, newfd)) == -1) {
	perror("dup2");
	exit(1);
    }
    return ret;
}


int main()
{
    int c2p[2];
    int errpipe[2];
    pid_t pid;
    char errpipe_buf[16];
    char buf[16394];
    ssize_t errpipe_count;
    size_t count;
    FILE *output;

    checked_pipe(c2p);
    checked_pipe(errpipe);
    fcntl(errpipe[1], F_SETFD, FD_CLOEXEC);

    pid = fork();
    switch (pid) {
    case -1:
	perror("fork");
	exit(1);
	break;

    case 0:
	/* child */
	checked_close(c2p[0]);
	checked_close(errpipe[0]);
	dup2(c2p[1], 1);
	dup2(c2p[1], 2);
	checked_close(c2p[1]);
	execlp("/bin/ls", "/bin/ls", "/etc/passwd", "/somebogus");
	/* If we are still around, exec failed */
	write(errpipe[1], "1", 1);
	_exit(255);
	break;

    default:
	/* parent */
	break;
    }

    checked_close(errpipe[1]);
    checked_close(c2p[1]);

    errpipe_count = read(errpipe[0], errpipe_buf, 1);
    switch (errpipe_count) {
    case -1:
	perror("read");
	exit(1);
	break;
    case 0:
	/* EOF, good */
	break;
    default:
	/* Our "1", exec failed */
	fprintf(stderr, "exec in child failed\n");
	exit(1);
	break;
    }

    /* now read output from child, via a FILE stream */
    if ((output = fdopen(c2p[0], "rb")) == NULL) {
	perror("fdopen");
	exit(1);
    }

    setvbuf(output, 0, _IONBF, 0);
    clearerr(output);
    count = fread(buf, 1, sizeof(buf), output);
    if (count >= 0) {
	fprintf(stderr, "Got %d bytes:\n", count);
	fprintf(stderr, "%s|", buf);
    }
    if (feof(output)) {
	fprintf(stderr, "EOF has been reached.\n");
    }
    if (ferror(output)) {
	fprintf(stderr, "An error occured.\n");
    }

    return 0;
}

/Peter Åstrand <astrand@lysator.liu.se>


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Igor Pechtchanski]

On Fri, 10 Dec 2004, Peter Astrand wrote:

> >> I've discovered that fread(ptr, size, nitems, stream) sometimes returns a
> >> value less than "nitems", but does not set feof() nor ferror(). As I
>
> >A simple, compilable test case with results shown on both linux and
> >cygwin would prove your theory.
>
> Here you go; example follows. Test run on Linux:
>
> ---
> Got 66 bytes:
> /bin/ls: /somebogus: Filen eller katalogen finns inte
> /etc/passwd
> |EOF has been reached.
> ---
>
> Test run on Cygwin:
>
> ---
> $ ./a.exe
> Got 9 bytes:
> /bin/ls: |
> ---
>
> As you can see, the result is partial, and there is no EOF nor error.
>
> I guess this is the problem described at
> http://sources.redhat.com/ml/newlib/2004/msg00477.html. Perhaps you can
> apply the suggested patch?

According to <http://sources.redhat.com/ml/newlib/2004/msg00478.html>, the
patch was checked in on Oct 26.  Cygwin uses the latest version of newlib.
Cygwin 1.5.12-1 was released on Nov 10, so it should contain this patch.

You can verify this by downloading the Cygwin source package, and looking
at newlib/libc/stdio/fread.c.
	Igor
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_		pechtcha@cs.nyu.edu
ZZZzz /,`.-'`'    -.  ;-;;,_		igor@watson.ibm.com
     |,4-  ) )-,_. ,\ (  `'-'		Igor Pechtchanski, Ph.D.
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

"The Sun will pass between the Earth and the Moon tonight for a total
Lunar eclipse..." -- WCBS Radio Newsbrief, Oct 27 2004, 12:01 pm EDT

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Christopher Faylor]

On Thu, Dec 09, 2004 at 07:49:18PM -0500, Igor Pechtchanski wrote:
>> I guess this is the problem described at
>> http://sources.redhat.com/ml/newlib/2004/msg00477.html. Perhaps you can
>> apply the suggested patch?
>
>According to <http://sources.redhat.com/ml/newlib/2004/msg00478.html>, the
>patch was checked in on Oct 26.  Cygwin uses the latest version of newlib.
>Cygwin 1.5.12-1 was released on Nov 10, so it should contain this patch.

And, from the release announcement:

- Fix return value for unbuffered fread.  (Jason Tishler)

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Peter Astrand]

On Thu, 9 Dec 2004, Igor Pechtchanski wrote:

> > >> I've discovered that fread(ptr, size, nitems, stream) sometimes returns a
> > >> value less than "nitems", but does not set feof() nor ferror(). As I
> >
> > >A simple, compilable test case with results shown on both linux and
> > >cygwin would prove your theory.
> >
> > Here you go; example follows. Test run on Linux:

> According to <http://sources.redhat.com/ml/newlib/2004/msg00478.html>, the
> patch was checked in on Oct 26.  Cygwin uses the latest version of newlib.
> Cygwin 1.5.12-1 was released on Nov 10, so it should contain this patch.
>
> You can verify this by downloading the Cygwin source package, and looking
> at newlib/libc/stdio/fread.c.

I've tested http://sources.redhat.com/ml/newlib/2004/txt00003.txt on
Cygwin, and it works. So, most probably, Cygwin uses the "fixed" newlib.


However, since my example code
(http://cygwin.com/ml/cygwin/2004-12/msg00305.html) still fails, something
is still wrong. Is it possible the the "fix" above actually caused this
problem?


/Peter Åstrand <astrand@lysator.liu.se>


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Jason Tishler]

Peter,

On Fri, Dec 10, 2004 at 11:18:12AM +0100, Peter Astrand wrote:
> However, since my example code
> (http://cygwin.com/ml/cygwin/2004-12/msg00305.html) still fails,
> something is still wrong. Is it possible the the "fix" above actually
> caused this problem?

AFAICT, the problems indicate by the following:

    http://python.org/sf/1071516

were already there before I submitted my unbuffered fread() newlib
patch.

Jason

-- 
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D  8784 1AFD E4CC ECF4 8EF6

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: short fread(), but no ferror/feof

[Dave Korn]

> -----Original Message-----
> From: cygwin-owner On Behalf Of Jason Tishler
> Sent: 10 December 2004 12:15

> Peter,
> 
> On Fri, Dec 10, 2004 at 11:18:12AM +0100, Peter Astrand wrote:
> > However, since my example code
> > (http://cygwin.com/ml/cygwin/2004-12/msg00305.html) still fails,
> > something is still wrong. Is it possible the the "fix" 
> above actually
> > caused this problem?
> 
> AFAICT, the problems indicate by the following:
> 
>     http://python.org/sf/1071516
> 
> were already there before I submitted my unbuffered fread() newlib
> patch.
> 
> Jason
 

  The fix and the problem Peter is seeing are orthogonal.

  The fix referred to above fixes this problem:

http://sources.redhat.com/ml/newlib/2004/msg00478.html
"Hence, one can see that fread() in unbuffered mode always returns the
specified count instead of the number of elements actually read."

  That is why, as Peter has observed, fread() returns the number of elements
actually read.  Without the patch, it would have been returning the entire
number requested, which would have been even wronger.

  However Peter's problem is that when fread() does a partial read, it doesn't
set either the EOF mark or the error indication on the file stream.  A strict
reading of the fread() specification suggests that it should have set one of
those.  OTOH, there isn't actually any error or EOF here.  It could possibly be
argued from the specified behaviour of read() and pipes that the error mark
should be set and that errno should be EAGAIN.  It may well be in practice that
errno does indeed have the value EAGAIN but fread() can't return -1 because
otherwise the application wouldn't know how much data it _had_ read.

  Of course, Peter can always detect when this situation has occurred, precisely
because fread() returns a value that, while >= 0, is < the number of elements
requested, and when feof() and ferror() both return zero, his code could deduce
that it's a short read from a pipe and try again.


    cheers, 
      DaveK
-- 
Can't think of a witty .sigline today....


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Peter Astrand]

Dave Korn wrote:

>  The fix and the problem Peter is seeing are orthogonal.
>
>  The fix referred to above fixes this problem:
>
>http://sources.redhat.com/ml/newlib/2004/msg00478.html
>"Hence, one can see that fread() in unbuffered mode always returns the
>specified count instead of the number of elements actually read."
>
>  That is why, as Peter has observed, fread() returns the number of
>elements
>actually read.  Without the patch, it would have been returning the entire
>number requested, which would have been even wronger.

Ah, thanks for clarifying this.


>  However Peter's problem is that when fread() does a partial read, it doesn't
>set either the EOF mark or the error indication on the file stream.  A strict
>reading of the fread() specification suggests that it should have set one of
>those.

IMO, there's no room for intepretation in this situation: As I stated
in my original post, IEEE Std 1003.1 is very clear:

"fread() shall return the number of elements successfully read which is
less than nitems only if a read error or end-of-file is encountered."


>OTOH, there isn't actually any error or EOF here.  It could possibly
>be argued from the specified behaviour of read() and pipes that the
>error mark should be set and that errno should be EAGAIN.  It may well
>be in practice that errno does indeed have the value EAGAIN but
>fread() can't return -1 because otherwise the application wouldn't
>know how much data it _had_ read.

I've discussed this issue with some local gurus, and the consensus is that
fread() should do as many read() calls as it takes. This is what glibc
does, for example (if I understand the source correctly). Basically,
fread() means "read everything". It's just plain wrong to return a partial
result, unless EOF or an error occured.


>  Of course, Peter can always detect when this situation has
>occurred, precisely because fread() returns a value that, while >= 0,
>is < the number of elements requested, and when feof() and ferror()
>both return zero, his code could deduce that it's a short read from a
>pipe and try again.

If you can convince the Python developers to add these checks to the
already-complicated fread() invocation in fileobject.c, I will stop
complaining... But fread() shouldn't behave like that.


(Please CC me on replies.)

/Peter Åstrand <astrand@lysator.liu.se>


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Christopher Faylor]

On Sat, Dec 11, 2004 at 11:32:07PM +0100, Peter Astrand wrote:
>I've discussed this issue with some local gurus, and the consensus is
>that fread() should do as many read() calls as it takes.  This is what
>glibc does, for example (if I understand the source correctly).
>Basically, fread() means "read everything".  It's just plain wrong to
>return a partial result, unless EOF or an error occured.
>
>>Of course, Peter can always detect when this situation has occurred,
>>precisely because fread() returns a value that, while >= 0, is < the
>>number of elements requested, and when feof() and ferror() both return
>>zero, his code could deduce that it's a short read from a pipe and try
>>again.
>
>If you can convince the Python developers to add these checks to the
>already-complicated fread() invocation in fileobject.c, I will stop
>complaining...  But fread() shouldn't behave like that.

Perhaps you should prevail upon your gurus to provide a patch to fix
this problem rather than relying on "complaining".

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Jeff Johnston]

Peter Astrand wrote:
> Dave Korn wrote:
> 
> 
>> The fix and the problem Peter is seeing are orthogonal.
>>
>> The fix referred to above fixes this problem:
>>
>>http://sources.redhat.com/ml/newlib/2004/msg00478.html
>>"Hence, one can see that fread() in unbuffered mode always returns the
>>specified count instead of the number of elements actually read."
>>
>> That is why, as Peter has observed, fread() returns the number of
>>elements
>>actually read.  Without the patch, it would have been returning the entire
>>number requested, which would have been even wronger.
> 
> 
> Ah, thanks for clarifying this.
> 
> 
> 
>> However Peter's problem is that when fread() does a partial read, it doesn't
>>set either the EOF mark or the error indication on the file stream.  A strict
>>reading of the fread() specification suggests that it should have set one of
>>those.
> 
> 
> IMO, there's no room for intepretation in this situation: As I stated
> in my original post, IEEE Std 1003.1 is very clear:
> 
> "fread() shall return the number of elements successfully read which is
> less than nitems only if a read error or end-of-file is encountered."
>

Where in the POSIX standard did you find that above line?  I could not locate it 
in mine.  The line above is in SUSV3.  ANSI and C99 are more relaxed and simply 
state that the number of elements returned may be less than asked if EOF or an 
error has occurred.

Regardless, the behavior is wrong.  The same test on a buffered file yields the EOF.

I have posted a patch.

-- Jeff J.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Peter Astrand]

On Mon, 13 Dec 2004, Jeff Johnston wrote:

> > "fread() shall return the number of elements successfully read which is
> > less than nitems only if a read error or end-of-file is encountered."
> >
>
> Where in the POSIX standard did you find that above line?  I could not locate it
> in mine.  The line above is in SUSV3.

The manpage fread(P) on Fedora Core 2. Is that SUSV3? The last line says
"POSIX", though.


> Regardless, the behavior is wrong.  The same test on a buffered file yields the EOF.
>
> I have posted a patch.

Great!


/Peter Åstrand <astrand@lysator.liu.se>


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Ralf Corsepius]

On Mon, 2004-12-13 at 21:53 +0100, Peter Astrand wrote:
> On Mon, 13 Dec 2004, Jeff Johnston wrote:
> 
> > > "fread() shall return the number of elements successfully read which is
> > > less than nitems only if a read error or end-of-file is encountered."
> > >
> >
> > Where in the POSIX standard did you find that above line?  I could not locate it
> > in mine.  The line above is in SUSV3.
> 
> The manpage fread(P) on Fedora Core 2. Is that SUSV3?

Single UNIX Specification, Version 3

http://www.opengroup.org/onlinepubs/009695399

Ralf




--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: short fread(), but no ferror/feof

[Jason Tishler]

Jeff,

On Mon, Dec 13, 2004 at 02:55:12PM -0500, Jeff Johnston wrote:
> Regardless, the behavior is wrong.  The same test on a buffered file
> yields the EOF.
> 
> I have posted a patch.

Thanks for the patch.  I have confirmed it resolves the following Cygwin
Python problem:

    http://sf.net/tracker/?func=detail&aid=1071516&group_id=5470&atid=105470

Jason

-- 
PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers
Fingerprint: 7A73 1405 7F2B E669 C19D  8784 1AFD E4CC ECF4 8EF6

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/