[Bug c++/21286] New: GNU extension stdio_filebuf problems when reading from pipe

[Bug c++/21286] New: GNU extension stdio_filebuf problems when reading from pipe

[ralfixx at gmx dot de]

Attach stdin (or any other FILE*) to an iostream via the rdbuf() created from a
FILE* by using the GNU extension stdio_filebuf() (see attached C++-File).

Input is just some big file with +11MB:

% wc -c big_file
11601584 big_file

This reads incorrect number of bytes when stdin is a pipe and using an block
size which is not some power of 2. The number of bytes read differs from run to run.
% cat big_file | ./t400 8687
total bytes read 9347072, blocks 1076
% cat big_file | ./t400 8687
total bytes read 978944, blocks 113
% cat big_file | ./t400 8687
total bytes read 8679424, blocks 1000
% cat big_file | ./t400 8687
total bytes read 3424256, blocks 395
% cat big_file | ./t400 8687
total bytes read 11601584, blocks 1336
% cat big_file | ./t400 2048
total bytes read 11601584, blocks 5665
% cat big_file | ./t400 2048
total bytes read 11601584, blocks 5665

No errors when stdin is redirected directly to the file.  

% ./t400 8676 < big_file
total bytes read 11601584, blocks 1338
% ./t400 8676 < big_file
total bytes read 11601584, blocks 1338
% ./t400 8676 < big_file
total bytes read 11601584, blocks 1338
% ./t400 8676 < big_file
total bytes read 11601584, blocks 1338

Also no problems when reading in 2048 or 4096 blocks:

% cat big_file | ./t400 2048
total bytes read 11601584, blocks 5665
% cat big_file | ./t400 2048
total bytes read 11601584, blocks 5665
% cat big_file | ./t400 2048
total bytes read 11601584, blocks 5665

% cat big_file | ./t400 4096
total bytes read 11601584, blocks 2833
% cat big_file | ./t400 4096
total bytes read 11601584, blocks 2833
% cat big_file | ./t400 4096
total bytes read 11601584, blocks 2833

Same problem on HP-UX 10.20 with gcc-4.0.0 (and 3.4.3 if it matters).

The same program runs w/o problems in gcc-3.4.1 and gcc-3.3.3 on the same platforms.

I actually had a shell loop trying block sizes from 1 to 10000 to find the
problematic ones, so if 8687 does not reproduce the bug, try the follwoing shell
loop:

i=8000
while test $i -lt 9000 ; do
  echo -e "$i: \\c"
  cat big_file | ./t400 $i
  i=`echo $i+1 | bc`
done


R'

-- 
           Summary: GNU extension stdio_filebuf problems when reading from
                    pipe
           Product: gcc
           Version: 4.0.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: c++
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: ralfixx at gmx dot de
                CC: gcc-bugs at gcc dot gnu dot org
 GCC build triplet: i686-pc-linux-gnu
  GCC host triplet: i686-pc-linux-gnu
GCC target triplet: i686-pc-linux-gnu


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug c++/21286] GNU extension stdio_filebuf problems when reading from pipe

[ralfixx at gmx dot de]

------- Additional Comments From ralfixx at gmx dot de  2005-04-29 15:56 -------
Created an attachment (id=8765)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=8765&action=view)
C++ source file to show the problem

g++ -o t400 t.cxx
usage: t400 [blocksize]

t400 reads all input and counts the bytes.  At EOF, a summary is printed.  The
total bytes read should match the input file size.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug c++/21286] GNU extension stdio_filebuf problems when reading from pipe

[ralfixx at gmx dot de]

------- Additional Comments From ralfixx at gmx dot de  2005-04-29 15:58 -------
(From update of attachment 8765)
The last paragraph in the attachted file is wrong, the bug is reproducable on
linux now.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug c++/21286] GNU extension stdio_filebuf problems when reading from pipe

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-04-29 17:28 -------
Indeed, this is easily explained considering that we have a new optimization
that issues larger underlying read sys-calls (have a look to xsgetn in fstream.tcc
to see what I mean): when everything goes well ;) are you noticing measurable
performance improvements? When pipes are involved, I'm not sure we can call this
a bug, because, when 27.6.1.3/28 talks about "end-of-file occurs on the input
sequence" it refers, in this case, not to the underlying file that you are piping
but to the pipe itself and, at that moment, indeed the pipe cannot make available
all the characters requested. I'd like to know your opinion, as a user: are you
noticing worthwhile performance improvements? Would you consider very annoying
trying to read again (calling clear()), when pipes are used? In case, disabling
the optimization to be safe would be very easy.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] GNU extension stdio_filebuf problems when reading from pipe

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-04-29 21:31 -------
Ok, I have asked privately Nathan Myers and what we need is rather simple: add
looping on short reads inside filebuf::sgetn, rather straightforward to
implement. Stay tuned!

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ncm at cantrip dot org
         AssignedTo|unassigned at gcc dot gnu   |pcarlini at suse dot de
                   |dot org                     |
             Status|WAITING                     |ASSIGNED


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[ncm-nospam at cantrip dot org]

------- Additional Comments From ncm-nospam at cantrip dot org  2005-04-30 02:11 -------
Just to add... if sgetn() loops reading until it gets n bytes, but 
underflow() accepts a pipe's short reads, then in_avail() will report 
the size of the short read.  Then, istream::read_some() will 
rdbuf->sgetn(p,rdbuf->in_avail()), thus matching POSIX pipe semantics 
when attached to a pipe.

(For those who don't know: as long as whoever is writing the
other end of the pipe writes no more than PIPE_BUF bytes into 
it, then POSIX requires that read() deliver the bytes on write 
boundaries.  Thus, if you're writing structs, the reader will 
see half a struct.  If you're writing short-enough lines, the 
reader will only ever see complete lines -- although you might
see more than one.  Watch out that PIPE_BUF is 4K or more 
on Linux, but only 512 bytes on BSD!)

It's exciting to finally get this right.  With a bit more work, 
I see now that we might be able to support nonblocking file 
descriptors, and maybe even zero-copy I/O, too!  Thank you, Ralf.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[ncm-nospam at cantrip dot org]

------- Additional Comments From ncm-nospam at cantrip dot org  2005-04-30 03:49 -------
(In reply to comment #5)
> ...  Thus, if you're writing structs, the reader will 
> see half a struct.  

Sorry, that's "will never see half a struct".



-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[cvs-commit at gcc dot gnu dot org]

------- Additional Comments From cvs-commit at gcc dot gnu dot org  2005-04-30 06:54 -------
Subject: Bug 21286

CVSROOT:	/cvs/gcc
Module name:	gcc
Changes by:	paolo@gcc.gnu.org	2005-04-30 06:54:08

Modified files:
	libstdc++-v3   : ChangeLog 
	libstdc++-v3/include/bits: fstream.tcc 

Log message:
	2005-04-29  Paolo Carlini  <pcarlini@suse.de>
	Nathan Myers  <ncm@cantrip.org>
	
	PR libstdc++/21286
	* include/bits/fstream.tcc (basic_filebuf<>::xsgetn):
	Loop on short reads; remove the work-around for
	libstdc++/20806, not needed anymore.

Patches:
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/ChangeLog.diff?cvsroot=gcc&r1=1.2986&r2=1.2987
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/include/bits/fstream.tcc.diff?cvsroot=gcc&r1=1.129&r2=1.130



-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |4.0.1


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[ralfixx at gmx dot de]

------- Additional Comments From ralfixx at gmx dot de  2005-04-30 14:16 -------
Paolo Carlini wrote:
> I'd like to know your opinion, as a user: are you
> noticing worthwhile performance improvements? 
> Would you consider very annoying trying to read again
> (calling clear()), when pipes are used? 

The issue seems already solved, but I'd like to add a comment to this question:
Yes, I would find it very annoying, since my program simply does not know
whether  a file descriptor is connected to a pipe or not.  Note that I have oly
checked for eof() im my program, so the above solution basically means: on
eof(), clear() and try to read again.  How often should I try this?  Etc.

Performance improvement or not, the stream should (must?) give the same results
regardless of the data source IMHO.  Performance can only be second here.

BTW, the program timing is as follows on my machine (reading from local file
system /tmp):

gcc 3.3.3: 
0.01user 0.01system 0:00.08elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k
0.01user 0.00system 0:00.08elapsed 23%CPU (0avgtext+0avgdata 0maxresident)k
0.01user 0.01system 0:00.11elapsed 25%CPU (0avgtext+0avgdata 0maxresident)k
0.01user 0.01system 0:00.08elapsed 27%CPU (0avgtext+0avgdata 0maxresident)k

gcc 4.0.0 (when reading all bytes from the pipe, not with the short reads)
0.00user 0.01system 0:00.04elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k
0.00user 0.01system 0:00.04elapsed 48%CPU (0avgtext+0avgdata 0maxresident)k
0.00user 0.01system 0:00.04elapsed 27%CPU (0avgtext+0avgdata 0maxresident)k
0.00user 0.01system 0:00.04elapsed 35%CPU (0avgtext+0avgdata 0maxresident)k

So yes, there seems to be a performance improvement (1.x CPU, half wall clock time).

Best regards
R'


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[ncm-nospam at cantrip dot org]

------- Additional Comments From ncm-nospam at cantrip dot org  2005-04-30 15:48 -------
A note for Ralf: It is incorrect to use cin.eof() to watch for the end
of a stream.  The correct flag to check is fail().  eof() is really 
meant for seeing if that's why op>> failed.  

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[cvs-commit at gcc dot gnu dot org]

------- Additional Comments From cvs-commit at gcc dot gnu dot org  2005-04-30 22:55 -------
Subject: Bug 21286

CVSROOT:	/cvs/gcc
Module name:	gcc
Branch: 	gcc-4_0-branch
Changes by:	paolo@gcc.gnu.org	2005-04-30 22:55:34

Modified files:
	libstdc++-v3   : ChangeLog 
	libstdc++-v3/include/bits: fstream.tcc 

Log message:
	2005-04-30  Paolo Carlini  <pcarlini@suse.de>
	Nathan Myers  <ncm@cantrip.org>
	
	PR libstdc++/21286
	* include/bits/fstream.tcc (basic_filebuf<>::xsgetn):
	Loop on short reads; remove the work-around for
	libstdc++/20806, not needed anymore.

Patches:
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/ChangeLog.diff?cvsroot=gcc&only_with_tag=gcc-4_0-branch&r1=1.2917.2.35&r2=1.2917.2.36
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/include/bits/fstream.tcc.diff?cvsroot=gcc&only_with_tag=gcc-4_0-branch&r1=1.128.10.1&r2=1.128.10.2



-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-04-30 22:57 -------
Should now be fixed for 4.1.0 and 4.0.1. Ralf, if you get a chance to test a
4.0.1 snapshot...

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[ralfixx at gmx dot de]

------- Additional Comments From ralfixx at gmx dot de  2005-05-01 17:44 -------
ncm-nospam@cantrip.org:
> A note for Ralf: It is incorrect to use cin.eof() 
> to watch for the end of a stream.  
> The correct flag to check is fail().  
> eof() is really meant for seeing if that's why op>> failed.  

Yup.  Thanks for pointing that out.  As for the 4.0.1 snapshot, is there a
specific place where to get it, or should I just use the CVS links on
http://www.gnu.org/software/gcc/cvs.html?

R'

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-05-01 18:12 -------
You can find weekly snapshots both of mainline and 4_0 here:

  ftp://gcc.gnu.org/pub/gcc/snapshots/

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[ralfixx at gmx dot de]

------- Additional Comments From ralfixx at gmx dot de  2005-05-01 19:12 -------
>  ftp://gcc.gnu.org/pub/gcc/snapshots/

Just tried gcc-4.0-20050430: the problem is gone.
However, I could not reintroduce the problem by just reverting the patch for
fstream.cc (still works after I undo the patch), but I think this is not
important.  Maybe there were other changes as well which prevent the problem
from manifesting in gcc-4.0-20050430.

Best regards and thanks for the quick solution.
R'

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-05-01 19:28 -------
OK, thanks. Maybe you cannot see the problem by just reverting the patch,
because at least part of the data remained in the cache: I found your procedure
very useful to expose the problem but very sensitive to that detail. If you can,
please keep on testing 4.0.1 prereleases: we really hate introducing regressions
between minor releases. Thanks.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[amu at alum dot mit dot edu]

------- Additional Comments From amu at alum dot mit dot edu  2005-07-14 20:45 -------
FWIW, the same broken optimization, and hence the same bug, also appears to have
materialized in the 3.4 branch somewhere between 3.4.2 and 3.4.3.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-07-14 23:51 -------
Thanks. At this point in 3_4-branch go only very-very safe changes. Therefore,
before considering fixing in that branch too the problem (*), let's test the
new algorithm in mainline and 4_0-branch for a while.

(*) By the way, I keep on wondering why people like so much the word "broken"
about software?!? Maybe because it's something immaterial, e.g., would not be
so happy to use the same term about a leg or an arm?!? ;)

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[ralfixx at gmx dot de]

------- Additional Comments From ralfixx at gmx dot de  2005-07-18 12:03 -------
> Thanks. At this point in 3_4-branch go only very-very safe changes. Therefore,
> before considering fixing in that branch too the problem (*), let's test the
> new algorithm in mainline and 4_0-branch for a while.

I'm wondering why this problem does not break tons of code, since all pipes are
basically broken (yes, broken: they don't supply all the data).  So each program
which runs as a filter should suffer from this problem.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-07-18 12:28 -------
> I'm wondering why this problem does not break tons of code,

Just read the audit trail, about that. In particular my comment #3: portable C++
code using pipes has better being very very careful with short reads, because
the current Standard is way too vague in this area (improvements in system
programming support are certainly planned for C++0x).

That said, the next 3.4.x is not scheduled for tomorrow, we shall fix it in time,
don't worry. By the way, I would suggest pronouncing "broken" using the same 
intonation of Martin Landau's Bela Lugosi in Ed Wood: "Beware, beware... broken,
broken..." ;)

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[ralfixx at gmx dot de]

------- Additional Comments From ralfixx at gmx dot de  2005-07-18 15:01 -------
> portable C++ code using pipes has better being very very 
> careful with short reads, because the current Standard is 
> way too vague in this area

I prefer C++ for the abstractions it allows.  Having to code my I/O with the
type of the underlying data stream type in mind defeats this abstraction to a
great deal...

> By the way, I would suggest pronouncing "broken" using the same 
> intonation of Martin Landau's Bela Lugosi in Ed Wood:
> "Beware, beware... broken, broken..." ;)

Let's pronounce it [kəˈpʊt] then :-)
http://en.wiktionary.org/wiki/Kaput


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-07-18 15:56 -------
> Let's pronounce it [kəˈpʊt] then :-)
> http://en.wiktionary.org/wiki/Kaput

Yes. My point was simply that in order to have the attention of the maintainers
you don't need to use exagerated expressions. Generally, technical explanations
are much better for that. Overstating the issue is more suited to make someone
nervous and, likely, obtain exactly the opposite effect.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [4.0/4.1 Regression] filebuf::xsgetn vs pipes

[cvs-commit at gcc dot gnu dot org]

------- Additional Comments From cvs-commit at gcc dot gnu dot org  2005-07-18 18:32 -------
Subject: Bug 21286

CVSROOT:	/cvs/gcc
Module name:	gcc
Branch: 	gcc-3_4-branch
Changes by:	paolo@gcc.gnu.org	2005-07-18 18:32:00

Modified files:
	libstdc++-v3   : ChangeLog 
	libstdc++-v3/include/bits: fstream.tcc 

Log message:
	2005-07-18  Paolo Carlini  <pcarlini@suse.de>
	Nathan Myers  <ncm@cantrip.org>
	
	PR libstdc++/21286
	* include/bits/fstream.tcc (basic_filebuf<>::xsgetn):
	Loop on short reads.

Patches:
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/ChangeLog.diff?cvsroot=gcc&only_with_tag=gcc-3_4-branch&r1=1.2224.2.227&r2=1.2224.2.228
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libstdc++-v3/include/bits/fstream.tcc.diff?cvsroot=gcc&only_with_tag=gcc-3_4-branch&r1=1.116.4.8&r2=1.116.4.9



-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [3.4/4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-07-18 18:39 -------
Fixed for 3.4.4 too.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[4.0/4.1 Regression]        |[3.4/4.0/4.1 Regression]
                   |filebuf::xsgetn vs pipes    |filebuf::xsgetn vs pipes
   Target Milestone|4.0.1                       |3.4.4


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [3.4/4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2005-07-18 18:40 -------
Sorry, I meant 3.4.5.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|3.4.4                       |4.0.1


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286

[Bug libstdc++/21286] [3.4/4.0/4.1 Regression] filebuf::xsgetn vs pipes

[pcarlini at suse dot de]

------- Comment #25 from pcarlini at suse dot de  2006-04-14 19:31 -------
*** Bug 27168 has been marked as a duplicate of this bug. ***


-- 

pcarlini at suse dot de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |hhinnant at apple dot com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21286