[Bug libstdc++/102259] New: ifstream::read(…, count) fails when count >= 2^31 on darwin

[Bug libstdc++/102259] New: ifstream::read(…, count) fails when count >= 2^31 on darwin

[mimomorin at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102259

            Bug ID: 102259
           Summary: ifstream::read(…, count) fails when count >= 2^31 on
                    darwin
           Product: gcc
           Version: 11.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libstdc++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: mimomorin at gmail dot com
  Target Milestone: ---

Created attachment 51431
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=51431&action=edit
Testcase for ifstream::read(…, count >= 2^31)

I tried to read a large file using `ifstream::read` on Mac, but it fails to
read any byte when count >= 2^31. Note that the system is 64-bit and
`std::streamsize` has 8 bytes.
Here is a testcase.

#include <iostream>
#include <fstream>
int main() {
    std::ifstream is{"2GB.bin", std::ios::binary}; // filesize >= 2^31 bytes
    auto buffer = new char[1LL << 31];
    is.read(buffer, 1LL << 31);
    std::cout << is.good() << " (" << is.gcount() << " bytes)\n";
    // Expected output: "1 (2147483648 bytes)"
    // Actual output (on Mac): "0 (0 bytes)"
}

My system is macOS 10.15 running on x86_64 Mac. The testcase failed on
Homebrew's GCC (ver. 6, 9, 10, 11) and MacPorts' GCC (ver. 6), but it succeeded
on LLVM Clang (trunk) and Apple Clang (ver. 12).

`ifstream::read(…, count)` works fine when count < 2^31. So if we split 
    is.read(buffer, 1LL << 31);
into
    is.read(buffer, (1LL << 31) - 1);
    is.read(buffer + (1LL << 31) - 1, 1);
then everything goes OK.
Additionally, `istringstream::read(…, count >= 2^31)` works fine both on GCC
and Clang.

I don't think this simple issue went unnoticed, so maybe I've missed something.

[Bug libstdc++/102259] ifstream::read(…, count) fails when count >= 2^31 on darwin

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102259

--- Comment #1 from Jonathan Wakely <redi at gcc dot gnu.org> ---
Well it works fine on other systems, which is probably why nobody noticed it.

I have no way to debug this.

[Bug libstdc++/102259] ifstream::read(…, count) fails when count >= 2^31 on darwin

[mimomorin at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102259

--- Comment #2 from Michel Morin <mimomorin at gmail dot com> ---
Whoa, darwin's (and FreeBSD's too?) `read(…, …, nbyte)` fails when nbyte >=
2^31! This is the culprit, I think. 

I also found the following description in FreeBSD's manpage of read
(https://www.unix.com/man-page/FreeBSD/2/read/):

    ERRORS
    [EINVAL] The value nbytes is greater than INT_MAX.

Given that the testcase works file when compiled with Clang, libcxx would have
some workround for it.

[Bug libstdc++/102259] ifstream::read(…, count) fails when count >= 2^31 on darwin

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102259

--- Comment #3 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
(In reply to Michel Morin from comment #2)
> Whoa, darwin's (and FreeBSD's too?) `read(…, …, nbyte)` fails when nbyte >=
> 2^31! This is the culprit, I think. 
> 
> I also found the following description in FreeBSD's manpage of read
> (https://www.unix.com/man-page/FreeBSD/2/read/):
> 
>     ERRORS
>     [EINVAL] The value nbytes is greater than INT_MAX.
> 
> Given that the testcase works file when compiled with Clang, libcxx would
> have some workround for it.

Maybe they do a loop around the read for sizes >= INT_MAX.  I don't know if
that is the correct thing to do or not.

[Bug libstdc++/102259] ifstream::read(…, count) fails when count >= 2^31 on darwin

[mimomorin at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102259

--- Comment #4 from Michel Morin <mimomorin at gmail dot com> ---
I googled and found that Rust and Python had the same issue (and fixed it): 
[Rust]
https://github.com/rust-lang/rust/issues/38590
(PR: https://github.com/ziglang/zig/pull/6333)
[Python]
https://bugs.python.org/issue24658
(PR: https://github.com/python/cpython/pull/1705)

These bug reports also says that the darwin's `write(…, …, nbyte)` fails when
nbyte > INT_MAX, and I confirmed that.

> Maybe they do a loop around the read for sizes >= INT_MAX.

Sounds good to me.

[Bug libstdc++/102259] ifstream::read(…, count) fails when count >= 2^31 on darwin

[mimomorin at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102259

--- Comment #5 from Michel Morin <mimomorin at gmail dot com> ---
I put a wrong link for Rust's PR. 
The correct link is https://github.com/rust-lang/rust/pull/38622 .

[Bug libstdc++/102259] ifstream::read(…, count) fails when count >= 2^31 on darwin

[redi at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102259

Jonathan Wakely <redi at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2021-09-13
             Status|UNCONFIRMED                 |NEW

--- Comment #6 from Jonathan Wakely <redi at gcc dot gnu.org> ---
Thanks for the pointers. I'll get around to adding that loop one day.