[Bug libc/12724] New: fclose violates POSIX 2008 on seekable input streams

[Bug libc/12724] New: fclose violates POSIX 2008 on seekable input streams

[eblake at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

           Summary: fclose violates POSIX 2008 on seekable input streams
           Product: glibc
           Version: 2.13
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper.fsp@gmail.com
        ReportedBy: eblake@redhat.com


POSIX 2008 states:

"If the file is not already at EOF, and the file is one capable of seeking, the
file offset of the underlying open file description shall be adjusted so that
the next operation on the open file description deals with the byte after the
last one read from or written to the stream being closed."

[XSH fclose,
http://pubs.opengroup.org/onlinepubs/9699919799/functions/fclose.html]

However, this sample program proves that for seekable input streams, glibc is
violating this requirement.  The same program runs to completion on Solaris 10.


#include <stdio.h>
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <assert.h>

#define NAME "test-fclose.t"

int
main (void)
{
  const char buf[] = "hello world";
  int fd;
  int fd2;
  FILE *f;

  /* Prepare a seekable file.  */
  fd = open (NAME, O_RDWR | O_CREAT | O_TRUNC, 0600);
  assert (0 <= fd);
  assert (write (fd, buf, sizeof buf) == sizeof buf);
  assert (lseek (fd, 1, SEEK_SET) == 1);

  /* Create an output stream visiting the file; when it is closed, all
     other file descriptors visiting the file must see the new file
     position.  */
  fd2 = dup (fd);
  assert (0 <= fd2);
  f = fdopen (fd2, "w");
  assert (f);
  assert (fputc (buf[1], f) == buf[1]);
  assert (fclose (f) == 0);
  errno = 0;
  assert (lseek (fd2, 0, SEEK_CUR) == -1);
  assert (errno == EBADF);
  assert (lseek (fd, 0, SEEK_CUR) == 2);

  /* Likewise for an input stream.  */
  fd2 = dup (fd);
  assert (0 <= fd2);
  f = fdopen (fd2, "r");
  assert (f);
  assert (fgetc (f) == buf[2]);
  assert (fclose (f) == 0);
  errno = 0;
  assert (lseek (fd2, 0, SEEK_CUR) == -1);
  assert (errno == EBADF);
  assert (lseek (fd, 0, SEEK_CUR) == 3);

  /* Clean up.  */
  assert (remove (NAME) == 0);

  return 0;
}

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12724] fclose violates POSIX 2008 on seekable input streams

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #1 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-05-14 01:11:00 UTC ---
Fixed in git.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12724] fclose violates POSIX 2008 on seekable input streams

[eblake at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Eric Blake <eblake at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |

--- Comment #2 from Eric Blake <eblake at redhat dot com> 2011-07-27 17:34:22 UTC ---
glibc 2.14 still has issues.  Per the latest wording:

http://austingroupbugs.net/view.php?id=87#c838

At page 805 line 26801 section fclose, change:

     the file offset of the underlying open file description shall be
     adjusted so that the next operation on the open file description
     deals with the byte after the last one read from or written to the
     stream being closed.

to:

     the file offset of the underlying open file description shall be
     set to the file position of the stream.

fclose() _must_ reposition the underlying fd, even if there was no last byte
read.  Therefore, this program, which passes on Solaris, shows that glibc is
still buggy:

#include <stdio.h>
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <assert.h>

#define NAME "test-fclose.t"

int
main (void)
{
  const char buf[] = "hello world";
  int fd;
  int fd2;
  FILE *f;

  /* Prepare a seekable file.  */
  fd = open (NAME, O_RDWR | O_CREAT | O_TRUNC, 0600);
  assert (0 <= fd);
  assert (write (fd, buf, sizeof buf) == sizeof buf);
  assert (lseek (fd, 1, SEEK_SET) == 1);

  /* Create an output stream visiting the file; when it is closed, all
     other file descriptors visiting the file must see the new file
     position.  */
  fd2 = dup (fd);
  assert (0 <= fd2);
  f = fdopen (fd2, "w");
  assert (f);
  assert(lseek(fd, 4, SEEK_SET) == 4);
  assert (fclose (f) == 0);
  errno = 0;
  assert (lseek (fd2, 0, SEEK_CUR) == -1);
  assert (errno == EBADF);
  assert (lseek (fd, 0, SEEK_CUR) == 4);

  /* Likewise for an input stream.  */
  fd2 = dup (fd);
  assert (0 <= fd2);
  f = fdopen (fd2, "r");
  assert (f);
  assert(lseek(fd, 4, SEEK_SET) == 4);
  assert (fclose (f) == 0);
  errno = 0;
  assert (lseek (fd2, 0, SEEK_CUR) == -1);
  assert (errno == EBADF);
  assert (lseek (fd, 0, SEEK_CUR) == 4);

  /* Clean up.  */
  assert (remove (NAME) == 0);

  return 0;
}

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12724] fclose violates POSIX 2008 on seekable input streams

[matz at suse dot de]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Michael Matz <matz at suse dot de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |matz at suse dot de

--- Comment #3 from Michael Matz <matz at suse dot de> 2011-08-12 15:31:21 UTC ---
Over at SuSE we ran into issues with this change.  See
  https://bugzilla.novell.com/show_bug.cgi?id=711829
.  Ruby does a dup/fdopen, but then doesn't do any reading/writing with 
the new stream, and when it gets fclosed the file position is reset also
for the other stream associated with the same file.  That one does
reads/writes.

I believe that's exactly the interaction between active/inactive streams
that http://austingroupbugs.net/view.php?id=87#c895 is talking about.

Now, that may or may not be useless code in ruby, but there we are.
The current half-POSIX-compliant way is worse than either the non-compliant
or the compliant way.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12724] fclose violates POSIX 2008 on seekable input streams

[vapier at gentoo dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Mike Frysinger <vapier at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |toolchain at gentoo dot org

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[jsm28 at gcc dot gnu.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Joseph Myers <jsm28 at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|libc                        |stdio

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[vapier at gentoo dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Mike Frysinger <vapier at gentoo dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |2.17

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[carlos_odonell at mentor dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Carlos O'Donell <carlos_odonell at mentor dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |UNCONFIRMED
                 CC|                            |carlos_odonell at mentor
                   |                            |dot com
         AssignedTo|drepper.fsp at gmail dot    |unassigned at sourceware
                   |com                         |dot org
     Ever Confirmed|1                           |0

--- Comment #4 from Carlos O'Donell <carlos_odonell at mentor dot com> 2012-11-29 15:19:04 UTC ---
We need someone to look into this on master and see if they can come up with a
patch to fix this.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[carlos_odonell at mentor dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Carlos O'Donell <carlos_odonell at mentor dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P2                          |P1
   Target Milestone|2.17                        |2.18

--- Comment #5 from Carlos O'Donell <carlos_odonell at mentor dot com> 2012-12-01 19:56:12 UTC ---
We are reverting this patch for 2.17 and will be fixing this correctly for 2.18
and ensuring that existing applications are not broken.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[carlos at systemhalted dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Carlos O'Donell <carlos at systemhalted dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|carlos_odonell at mentor    |carlos at systemhalted dot
                   |dot com                     |org

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[hannes at stressinduktion dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

Hannes Frederic Sowa <hannes at stressinduktion dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hannes at stressinduktion
                   |                            |dot org

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[eblake at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12724

--- Comment #6 from Eric Blake <eblake at redhat dot com> ---
Yet another example of how glibc violates POSIX:
http://austingroupbugs.net/view.php?id=816

and now cygwin has chosen to copy glibc's buggy behavior:
http://cygwin.com/ml/cygwin/2014-01/threads.html#00110

Any word on when this will be fixed?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[eblake at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12724

--- Comment #7 from Eric Blake <eblake at redhat dot com> ---
Intuitively, glibc is broken as long as:

$ seq 3 > file
$ { sed -n 1q; sed -n 1q; cat; } < file

has incorrect output.  On Solaris, where fclose (and fflush(NULL) and exit())
correctly sync the seekable position of the FILE* back to the underyling fd,
the above test has the POSIX-mandated behavior of acting like 'tail -n+3 file'.
 [It is possible to use gnulib's 'close-stdin' module to make 'sed' work around
glibc's bug, and in fact, coreutils already does so for all of its programs
which use stdin without reading to EOF; but we shouldn't have to work around
the bug in every single program that reads partial input from stdin when we can
instead fix it completely by fixing glibc]

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[joseph at codesourcery dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12724

--- Comment #8 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
On Fri, 17 Jan 2014, eblake at redhat dot com wrote:

> Any word on when this will be fixed?

When I started going through open "math" bugs a couple of years ago, 
fixing them and identifying underlying issues (with the implementation, 
test coverage, etc.) and other bugs in the course of fixing them, I was 
hoping for other people similarly to pick up the roles of glibc experts in 
particular areas, go through open bugs, fix them, file bugs for issues 
found, improve test coverage, understand the strengths and weaknesses of 
glibc in those areas and use that understanding to seek out and fix 
classes of bugs that seem likely to be present (without waiting for users 
to find and report them).

I still hope for people to pick up such roles.  One area needing an expert 
to take on leadership in resolving known issues is stdio.  Another is 
POSIX conformance, including going through clarifications / changes 
resulting from issues reported against various POSIX versions to ensure 
they are properly reflected in glibc, and bugs filed if not (and then, in 
future, to ensure bugs get filed for all future POSIX clarifications / 
changes indicating that a glibc change is needed).  Related to that is 
tracking new feature additions in POSIX (both between past revisions and 
in future ones) and ensuring we understand what features are missing from 
glibc.  Each component in Bugzilla could do with at least one expert.  
"libc" - miscellaneous bugs - needs a generalist who will, inter alia, 
seek consensus on questions of whether bugs that are really feature 
requests reflect features we do or do not want to have.

So: I'd expect this bug to be fixed after someone takes the lead on fixing 
it, possibly as part of taking the lead more generally on sorting out 
stdio or POSIX conformance issues, and including analysis of the 
compatibility issues and getting consensus on what might need doing for 
compatibility with existing binaries.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12724

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[eblake at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12724

Eric Blake <eblake at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends on|                            |12799


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=12799
[Bug 12799] fflush violates POSIX on seekable input streams
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug stdio/12724] fclose violates POSIX 2008 on seekable input streams

[eblake at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12724

--- Comment #9 from Eric Blake <eblake at redhat dot com> ---
Still buggy in 2020; fixing fflush(NULL) to reset the underlying offset of
seekable stdin [bug 12799] may be sufficient to fix this one as well.

-- 
You are receiving this mail because:
You are on the CC list for the bug.