[Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom

[Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

            Bug ID: 98129
           Summary: Failure on reading big chunk of /dev/urandom
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libfortran
          Assignee: unassigned at gcc dot gnu.org
          Reporter: tkoenig at gcc dot gnu.org
  Target Milestone: ---

The following program

program main
  implicit none
  integer, parameter :: n = 10**7
  integer :: u,v
  integer, dimension(:), allocatable :: a
  open (newunit=u,file="/dev/urandom",form="unformatted",access="stream")
  open (newunit=v,file="/dev/null",form="unformatted",access="stream")
  allocate (a(n))
  read (u) a
  write (v) a
end program main

fails on Linux with

At line 9 of file read.f90
Fortran runtime error: End of file

Error termination. Backtrace:
#0  0x7fa9c7be372f in read_block_direct
        at ../../../trunk/libgfortran/io/transfer.c:664
#1  0x7fa9c7be372f in unformatted_read
        at ../../../trunk/libgfortran/io/transfer.c:1127
#2  0x400bf3 in ???
#3  0x400c9c in ???
#4  0x7fa9c6e64349 in __libc_start_main
        at ../csu/libc-start.c:308
#5  0x400909 in ???
        at ../sysdeps/x86_64/start.S:120
#6  0xffffffffffffffff in ???

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2020-12-04

--- Comment #1 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
WORKSFORME on Darwin even Sith n=10**9 (7s).

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

--- Comment #2 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
The problem seems to be related to an early return from the read system call:

strace -e trace=open,read,close ./a.out
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\v\2\0\0\0\0\0"...,
832) = 832
close(3)                                = 0
close(3)                                = 0
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\217\0\0\0\0\0\0"...,
832) = 832
close(3)                                = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260/\0\0\0\0\0\0"...,
832) = 832
close(3)                                = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@*\0\0\0\0\0\0"..., 832)
= 832
close(3)                                = 0
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`D\2\0\0\0\0\0"..., 832)
= 832
close(3)                                = 0
read(3,
":\312\275\302\373|\204[c`\20\275\230\205\326@\255Mj\263-qd\336\30\300\2G\326\215\333J"...,
40000000) = 33554431
At line 9 of file read.f90
Fortran runtime error: End of file

Error termination. Backtrace:
close(5)                                = 0
close(5)                                = 0
close(6)                                = 0
close(5)                                = 0
close(5)                                = 0
close(5)                                = 0
close(6)                                = 0
close(5)                                = 0
close(6)                                = 0
close(5)                                = 0
#0  0x7fd9847c572f in read_block_direct
        at ../../../trunk/libgfortran/io/transfer.c:664
#1  0x7fd9847c572f in unformatted_read
        at ../../../trunk/libgfortran/io/transfer.c:1127
#2  0x400bf3 in ???
#3  0x400c9c in ???
#4  0x7fd983a46349 in __libc_start_main
        at ../csu/libc-start.c:308
#5  0x400909 in ???
        at ../sysdeps/x86_64/start.S:120
#6  0xffffffffffffffff in ???
close(4)                                = 0
close(3)                                = 0
+++ exited with 2 +++

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

--- Comment #3 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
The problem seems to be that we assume that a short read is always
an EOF, in read_block_direct:

      if (unlikely ((ssize_t) nbytes != have_read_record))
        {
          /* Short read,  e.g. if we hit EOF.  For stream files,
           we have to set the end-of-file condition.  */
          hit_eof (dtp);
        }
      return;
    }

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[kargl at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

kargl at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kargl at gcc dot gnu.org

--- Comment #4 from kargl at gcc dot gnu.org ---
WORKSFORME on i386-*-freebsd with 'n = 10**m' and m = 1, 2, 3, ..., 8.  On this
system with ' n = 10**9', I get

mobile:kargl[248] gfcx -o z -O a.f90 && ./z
In file 'a.f90', around line 8: Error allocating 4000000000 bytes: Cannot
allocate memory

So, it looks target specific.  Please change the component libgfortran to
whatever target you are using.

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Target|                            |x86_64-pc-linux-gnu

--- Comment #5 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
Question... on your respective systems, could you strace or truss it and find
if there is a short read?

On Linux, there seems to be a limitation of how many bytes
a read from /dev/urandom returns, and we assume that this is
an end of file.

However, this is not correct - we can only safely assume eof if
read() returns zero bytes.

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[kargl at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

kargl at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kargl at gcc dot gnu.org

--- Comment #6 from kargl at gcc dot gnu.org ---
(In reply to Thomas Koenig from comment #5)
> Question... on your respective systems, could you strace or truss it and find
> if there is a short read?
> 
> On Linux, there seems to be a limitation of how many bytes
> a read from /dev/urandom returns, and we assume that this is
> an end of file.
> 
> However, this is not correct - we can only safely assume eof if
> read() returns zero bytes.

Well, the test completes so I would expect that there isn't a short
read.  I modified the program to write to 'txt.dat' and get the 
expected file sizes for various 'n = 10**m' values.  FreeBSD does
not have strace, but ktrace suggests that the read/write occur
as expected.

Looking at the FreeBSD manpage, I find

  The /dev/urandom interface returns bytes regardless of the amount of
  entropy available.  It does not block on a read request due to lack
  of entropy.

Does urandom block on your system, and is this interpreted as a short
read?

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

anlauf at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |anlauf at gcc dot gnu.org

--- Comment #7 from anlauf at gcc dot gnu.org ---
Intel and PGI/Nvidia work w/o problems on my Linux system.
We might try to solve it in a similar way.

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

--- Comment #8 from anlauf at gcc dot gnu.org ---
Created attachment 49687
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=49687&action=edit
Untested patch (proof of concept)

Here's a possible patch that retries after short reads.

Not regtested.

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

anlauf at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |NEW

--- Comment #9 from anlauf at gcc dot gnu.org ---
The patch seems to regtest ok, but certainly needs some wider testing.

NIST?  Other I/O test suites?

May also need some cosmetic cleanup to match GNU coding style.

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

--- Comment #10 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
(In reply to anlauf from comment #9)
> The patch seems to regtest ok, but certainly needs some wider testing.

Actually, I think the bug is in io/unix.c:raw_read. That should take
care of repeating the reads as needed.

Seems like that, if nbyte <= MAX_CHUNK, we do not take account of the
possibility of a short read.

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

--- Comment #11 from anlauf at gcc dot gnu.org ---
(In reply to Thomas Koenig from comment #10)
> Seems like that, if nbyte <= MAX_CHUNK, we do not take account of the
> possibility of a short read.

Yes, that seems to be the better/right place.

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

anlauf at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #49687|0                           |1
        is obsolete|                            |

--- Comment #12 from anlauf at gcc dot gnu.org ---
Created attachment 50025
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=50025&action=edit
Revised patch, addressing the issue in unix.c

Hi Thomas,

I've regtested the attached patch.  It removes the first (problematic) part
of raw_read, and fixes the second part.  It seems to regtest fine, BUT:

I had one (random?) failure with append_1.f90 which I am unable to reproduce.

Looking at the strace output, it appears that there is one additional read(2)
with the patch that returns 0.  This is reproducible.  I have not yet found
out why this happens, but then I am probably to blind-sighted.

But it fixes the testcase.

[Bug libfortran/98129] Failure on reading big chunk of /dev/urandom

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129

--- Comment #13 from anlauf at gcc dot gnu.org ---
(In reply to anlauf from comment #12)
> I had one (random?) failure with append_1.f90 which I am unable to reproduce.

Likely origin: parallel make -j4 check-fortran and

% grep -il open.*[\"\']foo[\'\"] gcc/testsuite/gfortran.dg/ -r
gcc/testsuite/gfortran.dg/append_1.f90
gcc/testsuite/gfortran.dg/direct_io_2.f90
gcc/testsuite/gfortran.dg/io_constraints_7.f03
gcc/testsuite/gfortran.dg/large_unit_2.f90
gcc/testsuite/gfortran.dg/open_access_1.f90
gcc/testsuite/gfortran.dg/open_access_append_1.f90
gcc/testsuite/gfortran.dg/open_status_1.f90
gcc/testsuite/gfortran.dg/pr20163-2.f

The additional read seen in the strace output comes from the first read
attempt on the file which is shorter (169 bytes) that the buffer (8k),
so there is another read attempt (returning 0) because of end of file.
So that is a feature of the present fix.