[Bug libstdc++/63746] New: i/o fragmentation inside basic_filebuf::xsputn

[Bug libstdc++/63746] New: i/o fragmentation inside basic_filebuf::xsputn

[pluto at agmk dot net]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63746

            Bug ID: 63746
           Summary: i/o fragmentation inside basic_filebuf::xsputn
           Product: gcc
           Version: 4.9.2
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libstdc++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: pluto at agmk dot net

Hi,
i'm trying to minimize small i/o operations for better overal
application perfromance over networked filesystems.
setting quite large stream buffer doesn't work as axpected
in some cases.

#include <iostream>
#include <fstream>
#include <memory>
#include <sstream>
#include <string>

int main()
{
    std::size_t bufferSize = 1024*1024;
    std::unique_ptr< char > buffer( new char[ bufferSize ] );
    std::ofstream f;
    f.rdbuf()->pubsetbuf( buffer.get(), bufferSize );
    f.open( "s.txt", std::ios_base::out | std::ios_base::binary );

    std::stringstream ss;

    std::string s1( 10240, 'x' );

    ss.str( std::string() );
    ss << s1;
    f << ss.rdbuf();

    ss.str( std::string() );
    ss << s1;
    f << ss.rdbuf();

    f.close();
}



% g++ s.cpp -o s -Wall -g2 -std=c++11
% LANG=C strace ./s
(...)
open("s.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
writev(3, [{NULL, 0}, {"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 8136}], 2) = 8136
writev(3, [{"", 0}, {"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 2104}], 2) = 2104
writev(3, [{"", 0}, {"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 10240}], 2) = 10240
close(3)


i've expected that 20kB i/o from testcase will fit in the 1MB buffer
but the fstream.tcc contains magic const __chunk = 1k and logic that
not use the buffer in the optimal way.

[Bug libstdc++/63746] i/o fragmentation inside basic_filebuf::xsputn

[pluto at agmk dot net]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63746

--- Comment #1 from Pawel Sikora <pluto at agmk dot net> ---
Created attachment 33893
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=33893&action=edit
my hack.

[Bug libstdc++/63746] i/o fragmentation inside basic_filebuf::xsputn

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63746

Jonathan Wakely <redi at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at gcc dot gnu.org      |redi at gcc dot gnu.org
             Status|NEW                         |ASSIGNED
   Target Milestone|---                         |13.0

--- Comment #4 from Jonathan Wakely <redi at gcc dot gnu.org> ---
See https://gcc.gnu.org/pipermail/gcc-patches/2022-September/601082.html for a
new patch submission (making the same change) with a reproducer and benchmarks.

[Bug libstdc++/63746] i/o fragmentation inside basic_filebuf::xsputn

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63746

Jonathan Wakely <redi at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=11378

--- Comment #5 from Jonathan Wakely <redi at gcc dot gnu.org> ---
For the record, the change to bypass the buffer for large writes (including
anything >= 1024) was done for PR 11378.

[Bug libstdc++/63746] i/o fragmentation inside basic_filebuf::xsputn

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63746

--- Comment #6 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jonathan Wakely <redi@gcc.gnu.org>:

https://gcc.gnu.org/g:3f1519eef5cbdcea2f18445852f5482798e3936a

commit r13-3815-g3f1519eef5cbdcea2f18445852f5482798e3936a
Author: Charles-François Natali <cf.natali@gmail.com>
Date:   Thu Oct 6 20:02:56 2022 +0100

    libstdc++: basic_filebuf: don't flush more often than necessary [PR63746]

    `basic_filebuf::xsputn` would bypass the buffer when passed a chunk of
    size 1024 and above, seemingly as an optimisation.

    This can have a significant performance impact if the overhead of a
    `write` syscall is non-negligible, e.g. on a slow disk, on network
    filesystems, or simply during IO contention because instead of flushing
    every `BUFSIZ` (by default), we can flush every 1024 char.
    The impact is even greater with custom larger buffers, e.g. for network
    filesystems, because the code could issue `write` for example 1000X more
    often than necessary with respect to the buffer size.
    It also introduces a significant discontinuity in performance when
    writing chunks of size 1024 and above.

    Instead, it makes sense to only bypass the buffer if the amount of data
    to be written is larger than the buffer capacity.

    Signed-off-by: Charles-Francois Natali <cf.natali@gmail.com>

    libstdc++-v3/ChangeLog:

            PR libstdc++/63746
            * include/bits/fstream.tcc (basic_filbuf::xsputn): Remove
            1024-byte chunking that bypasses the buffer for large writes.
            * testsuite/27_io/basic_filebuf/sputn/char/63746.cc: New test.

[Bug libstdc++/63746] i/o fragmentation inside basic_filebuf::xsputn

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63746

Jonathan Wakely <redi at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #7 from Jonathan Wakely <redi at gcc dot gnu.org> ---
Done for GCC 13

[Bug libstdc++/63746] i/o fragmentation inside basic_filebuf::xsputn

[pawel_sikora at zoho dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63746

--- Comment #8 from Pawel Sikora <pawel_sikora at zoho dot com> ---
hi,

nice to see a final gcc fix.

gl&br, pawel.