[Bug libstdc++/20806] New: basic_filebuf::xsgetn() fails with text mode and

[Bug libstdc++/20806] New: basic_filebuf::xsgetn() fails with text mode and

[dannysmith at users dot sourceforge dot net]

 

-- 
           Summary: basic_filebuf::xsgetn()  fails with text mode and
           Product: gcc
           Version: 4.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libstdc++
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: dannysmith at users dot sourceforge dot net
                CC: gcc-bugs at gcc dot gnu dot org
 GCC build triplet: i686-pc-mingw32
  GCC host triplet: i686-pc-mingw32
GCC target triplet: i686-pc-mingw32


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[dannysmith at users dot sourceforge dot net]

------- Additional Comments From dannysmith at users dot sourceforge dot net  2005-04-07 08:35 -------

This optimization in basic_filebuf::xsgetn() causes problems on DOS based file 
sytstem when ifstreams are opened in text mode (\r\n line endings) and the 
user suppled buffer length exceeds _M_buf_size. 

2004-09-13  Paolo Carlini  <pcarlini@suse.de>

	PR libstdc++/11722
	* include/std/std_fstream.h (xsgetn): Declare only.
	* include/bits/fstream.tcc (xsgetn): Define, optimize for the
	always_noconv() case: when __n > __buflen, copy the available
	buffer and issue a direct read.

The problem is that the native read translates the input \r\n to
\n,  returning the number of chars written, so this test:

fstream.tcc:line 550 
	   if (__len == __n)
	     {
	       _M_set_buffer(0);
	       _M_reading = true;
	     }
 
fails.

Attached is a testcase submitted by a mingw user, The input file,
c:\winnt\schedlog.txt, has DOS line endings and is 32667 bytes in size.
The testcase stops after the first read with:

Is EOF? 1	gcount: 494

The first 512 bytes contain 18 line endings  

After reverting the xsgetn patch, or disabling for test mode files
with:

*** fstream.tcc.orig	Sun Jan 30 17:44:23 2005
--- fstream.tcc	Wed Apr  6 21:48:11 2005
*************** namespace std
*** 521,530 ****
         // future: when __n > __buflen we read directly instead of using the
         // buffer repeatedly.
         const bool __testin = _M_mode & ios_base::in;
         const streamsize __buflen = _M_buf_size > 1 ? _M_buf_size - 1
  	                                                 : 1;
         if (__n > __buflen && __check_facet(_M_codecvt).always_noconv()
! 	   && __testin && !_M_writing)
  	 {
  	   // First, copy the chars already present in the buffer.
  	   const streamsize __avail = this->egptr() - this->gptr();
--- 521,531 ----
         // future: when __n > __buflen we read directly instead of using the
         // buffer repeatedly.
         const bool __testin = _M_mode & ios_base::in;
+        const bool __testbinary = _M_mode & ios_base::binary;
         const streamsize __buflen = _M_buf_size > 1 ? _M_buf_size - 1
  	                                                 : 1;
         if (__n > __buflen && __check_facet(_M_codecvt).always_noconv()
! 	   && __testin && __testbinary && !_M_writing)
  	 {
  	   // First, copy the chars already present in the buffer.
  	   const streamsize __avail = this->egptr() - this->gptr();

the testcase produces:

Is EOF? 0	gcount: 512
Is EOF? 0	gcount: 512

 <snip  57 reads of 512 bytes > 

Is EOF? 0	gcount: 512
Is EOF? 1	gcount: 294


Disabling this optimization for non-binary input streams on all platforms is a
bit extreme.   Should this be conditional on an os_defines.h define?

Danny


Testcase modified from:
https://sourceforge.net/tracker/?
func=detail&atid=102435&aid=1171379&group_id=2435

#include <fstream>
using namespace std;

#define BS 512

int main (void)
{
  char buf [BS+1];
  // change to any DOS text file > BS bytes
  ifstream f ("c:\\winnt\\schedlog.txt") ;
  int r;
  while (!f.eof ())
   {
     f.read (buf, BS);
     r = f.gcount ();
     buf [r] = 0;

    fprintf (stderr, "%d %d: %s\n", f.eof (), r, buf);
   }

return 0;
}


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|basic_filebuf::xsgetn()     |basic_filebuf::xsgetn()
                   |fails with text mode and    |fails with text mode and DOS
                   |                            |line endings and large
                   |                            |buffers


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pinskia at gcc dot gnu dot org]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|basic_filebuf::xsgetn()     |[3.4/4.0/4.1 Regression]
                   |fails with text mode and DOS|basic_filebuf::xsgetn()
                   |line endings and large      |fails with text mode and DOS
                   |buffers                     |line endings and large
                   |                            |buffers


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-04-07 09:14 -------
> The problem is that the native read translates the input \r\n to
> \n,  returning the number of chars written,

This is a very fundamental assumption to break, which is likely to cause much
more problems elsewhere, maybe only latent now. The correct solution cannot
be just working around it in this function, instead something like using
unconditionally binary mode on mingw + libstdc++-v3. Ideas?

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-04-07 09:14 -------
> The problem is that the native read translates the input \r\n to
> \n,  returning the number of chars written,

This is a very fundamental assumption to break, which is likely to case much
more problems elsewhere, maybe only latent now. The correct solution cannot
be just working around it in this function, instead something like using
unconditionally binary mode on mingw + libstdc++-v3. Ideas?

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pcarlini at suse dot de]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pcarlini at suse dot de


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-04-07 10:31 -------
Ok, now I see the real, underlying problem: in config/io/basic_file_stdio.cc we
don't deal properly with "short read", that is, we don't try to loop if the
number of read chars is < n but > 0. Fixing that should also fix this specific
issue, but seems a bit risky for 4.0 :( I'm going to prepare a patch, anyway.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|unassigned at gcc dot gnu   |pcarlini at suse dot de
                   |dot org                     |
             Status|UNCONFIRMED                 |ASSIGNED
     Ever Confirmed|                            |1
   Last reconfirmed|0000-00-00 00:00:00         |2005-04-07 10:31:03
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-04-07 13:02 -------
Actually, we are not already doing that for a reason: doesn't work with pipes
(fifos), because, after the first succesful, short read, they block...

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[cvs-commit at gcc dot gnu dot org]

------- Additional Comments From cvs-commit at gcc dot gnu dot org  2005-04-08 17:31 -------
Subject: Bug 20806

CVSROOT:	/cvs/gcc
Module name:	gcc
Changes by:	paolo@gcc.gnu.org	2005-04-08 17:31:34

Modified files:
	libstdc++-v3   : ChangeLog 
	libstdc++-v3/include/bits: fstream.tcc 
	libstdc++-v3/config/os/mingw32: os_defines.h 
	libstdc++-v3/config/os/newlib: os_defines.h 

Log message:
	2005-04-08  Danny Smith  <dannysmith@users.sourceforge.net>
	Paolo Carlini  <pcarlini@suse.de>
	
	PR libstdc++/20806
	* config/os/mingw32/os_defines.h: Define
	_GLIBCXX_HAVE_DOS_BASED_FILESYSTEM.
	* config/os/newlib/os_defines.h: Likewise, for __CYGWIN__.
	* include/bits/fstream.tcc (basic_filebuf<>::showmanyc()):
	Use it.
	(basic_filebuf<>::xsgetn(_CharT*, streamsize)): Likewise.

Patches:
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/ChangeLog.diff?cvsroot=gcc&r1=1.2964&r2=1.2965
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/include/bits/fstream.tcc.diff?cvsroot=gcc&r1=1.128&r2=1.129
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/config/os/mingw32/os_defines.h.diff?cvsroot=gcc&r1=1.3&r2=1.4
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/config/os/newlib/os_defines.h.diff?cvsroot=gcc&r1=1.3&r2=1.4



-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[cvs-commit at gcc dot gnu dot org]

------- Additional Comments From cvs-commit at gcc dot gnu dot org  2005-04-08 17:34 -------
Subject: Bug 20806

CVSROOT:	/cvs/gcc
Module name:	gcc
Branch: 	gcc-4_0-branch
Changes by:	paolo@gcc.gnu.org	2005-04-08 17:34:35

Modified files:
	libstdc++-v3   : ChangeLog 
	libstdc++-v3/include/bits: fstream.tcc 
	libstdc++-v3/config/os/mingw32: os_defines.h 
	libstdc++-v3/config/os/newlib: os_defines.h 

Log message:
	2005-04-08  Danny Smith  <dannysmith@users.sourceforge.net>
	Paolo Carlini  <pcarlini@suse.de>
	
	PR libstdc++/20806
	* config/os/mingw32/os_defines.h: Define
	_GLIBCXX_HAVE_DOS_BASED_FILESYSTEM.
	* config/os/newlib/os_defines.h: Likewise, for __CYGWIN__.
	* include/bits/fstream.tcc (basic_filebuf<>::showmanyc()):
	Use it.
	(basic_filebuf<>::xsgetn(_CharT*, streamsize)): Likewise.

Patches:
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/ChangeLog.diff?cvsroot=gcc&only_with_tag=gcc-4_0-branch&r1=1.2917.2.20&r2=1.2917.2.21
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/include/bits/fstream.tcc.diff?cvsroot=gcc&only_with_tag=gcc-4_0-branch&r1=1.128&r2=1.128.10.1
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/config/os/mingw32/os_defines.h.diff?cvsroot=gcc&only_with_tag=gcc-4_0-branch&r1=1.3&r2=1.3.78.1
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/config/os/newlib/os_defines.h.diff?cvsroot=gcc&only_with_tag=gcc-4_0-branch&r1=1.3&r2=1.3.10.1



-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pcarlini at suse dot de]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |4.0.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[mmitchel at gcc dot gnu dot org]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.0.0                       |4.0.1


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pcarlini at suse dot de]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.0.1                       |4.0.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806

[Bug libstdc++/20806] [3.4/4.0/4.1 Regression] basic_filebuf::xsgetn() fails with text mode and DOS line endings and large buffers

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2005-04-25 00:32 -------
Fixed.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20806