* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
@ 2020-12-04 10:41 ` dominiq at lps dot ens.fr
2020-12-04 12:40 ` tkoenig at gcc dot gnu.org
` (11 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: dominiq at lps dot ens.fr @ 2020-12-04 10:41 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
Dominique d'Humieres <dominiq at lps dot ens.fr> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |WAITING
Ever confirmed|0 |1
Last reconfirmed| |2020-12-04
--- Comment #1 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
WORKSFORME on Darwin even Sith n=10**9 (7s).
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
2020-12-04 10:41 ` [Bug libfortran/98129] " dominiq at lps dot ens.fr
@ 2020-12-04 12:40 ` tkoenig at gcc dot gnu.org
2020-12-04 12:50 ` tkoenig at gcc dot gnu.org
` (10 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: tkoenig at gcc dot gnu.org @ 2020-12-04 12:40 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
--- Comment #2 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
The problem seems to be related to an early return from the read system call:
strace -e trace=open,read,close ./a.out
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\v\2\0\0\0\0\0"...,
832) = 832
close(3) = 0
close(3) = 0
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\217\0\0\0\0\0\0"...,
832) = 832
close(3) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260/\0\0\0\0\0\0"...,
832) = 832
close(3) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@*\0\0\0\0\0\0"..., 832)
= 832
close(3) = 0
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`D\2\0\0\0\0\0"..., 832)
= 832
close(3) = 0
read(3,
":\312\275\302\373|\204[c`\20\275\230\205\326@\255Mj\263-qd\336\30\300\2G\326\215\333J"...,
40000000) = 33554431
At line 9 of file read.f90
Fortran runtime error: End of file
Error termination. Backtrace:
close(5) = 0
close(5) = 0
close(6) = 0
close(5) = 0
close(5) = 0
close(5) = 0
close(6) = 0
close(5) = 0
close(6) = 0
close(5) = 0
#0 0x7fd9847c572f in read_block_direct
at ../../../trunk/libgfortran/io/transfer.c:664
#1 0x7fd9847c572f in unformatted_read
at ../../../trunk/libgfortran/io/transfer.c:1127
#2 0x400bf3 in ???
#3 0x400c9c in ???
#4 0x7fd983a46349 in __libc_start_main
at ../csu/libc-start.c:308
#5 0x400909 in ???
at ../sysdeps/x86_64/start.S:120
#6 0xffffffffffffffff in ???
close(4) = 0
close(3) = 0
+++ exited with 2 +++
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
2020-12-04 10:41 ` [Bug libfortran/98129] " dominiq at lps dot ens.fr
2020-12-04 12:40 ` tkoenig at gcc dot gnu.org
@ 2020-12-04 12:50 ` tkoenig at gcc dot gnu.org
2020-12-04 15:27 ` kargl at gcc dot gnu.org
` (9 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: tkoenig at gcc dot gnu.org @ 2020-12-04 12:50 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
--- Comment #3 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
The problem seems to be that we assume that a short read is always
an EOF, in read_block_direct:
if (unlikely ((ssize_t) nbytes != have_read_record))
{
/* Short read, e.g. if we hit EOF. For stream files,
we have to set the end-of-file condition. */
hit_eof (dtp);
}
return;
}
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (2 preceding siblings ...)
2020-12-04 12:50 ` tkoenig at gcc dot gnu.org
@ 2020-12-04 15:27 ` kargl at gcc dot gnu.org
2020-12-04 16:17 ` tkoenig at gcc dot gnu.org
` (8 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: kargl at gcc dot gnu.org @ 2020-12-04 15:27 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
kargl at gcc dot gnu.org changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |kargl at gcc dot gnu.org
--- Comment #4 from kargl at gcc dot gnu.org ---
WORKSFORME on i386-*-freebsd with 'n = 10**m' and m = 1, 2, 3, ..., 8. On this
system with ' n = 10**9', I get
mobile:kargl[248] gfcx -o z -O a.f90 && ./z
In file 'a.f90', around line 8: Error allocating 4000000000 bytes: Cannot
allocate memory
So, it looks target specific. Please change the component libgfortran to
whatever target you are using.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (3 preceding siblings ...)
2020-12-04 15:27 ` kargl at gcc dot gnu.org
@ 2020-12-04 16:17 ` tkoenig at gcc dot gnu.org
2020-12-04 18:16 ` kargl at gcc dot gnu.org
` (7 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: tkoenig at gcc dot gnu.org @ 2020-12-04 16:17 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
Thomas Koenig <tkoenig at gcc dot gnu.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Target| |x86_64-pc-linux-gnu
--- Comment #5 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
Question... on your respective systems, could you strace or truss it and find
if there is a short read?
On Linux, there seems to be a limitation of how many bytes
a read from /dev/urandom returns, and we assume that this is
an end of file.
However, this is not correct - we can only safely assume eof if
read() returns zero bytes.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (4 preceding siblings ...)
2020-12-04 16:17 ` tkoenig at gcc dot gnu.org
@ 2020-12-04 18:16 ` kargl at gcc dot gnu.org
2020-12-04 18:37 ` anlauf at gcc dot gnu.org
` (6 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: kargl at gcc dot gnu.org @ 2020-12-04 18:16 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
kargl at gcc dot gnu.org changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |kargl at gcc dot gnu.org
--- Comment #6 from kargl at gcc dot gnu.org ---
(In reply to Thomas Koenig from comment #5)
> Question... on your respective systems, could you strace or truss it and find
> if there is a short read?
>
> On Linux, there seems to be a limitation of how many bytes
> a read from /dev/urandom returns, and we assume that this is
> an end of file.
>
> However, this is not correct - we can only safely assume eof if
> read() returns zero bytes.
Well, the test completes so I would expect that there isn't a short
read. I modified the program to write to 'txt.dat' and get the
expected file sizes for various 'n = 10**m' values. FreeBSD does
not have strace, but ktrace suggests that the read/write occur
as expected.
Looking at the FreeBSD manpage, I find
The /dev/urandom interface returns bytes regardless of the amount of
entropy available. It does not block on a read request due to lack
of entropy.
Does urandom block on your system, and is this interpreted as a short
read?
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (5 preceding siblings ...)
2020-12-04 18:16 ` kargl at gcc dot gnu.org
@ 2020-12-04 18:37 ` anlauf at gcc dot gnu.org
2020-12-04 21:32 ` anlauf at gcc dot gnu.org
` (5 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: anlauf at gcc dot gnu.org @ 2020-12-04 18:37 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
anlauf at gcc dot gnu.org changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |anlauf at gcc dot gnu.org
--- Comment #7 from anlauf at gcc dot gnu.org ---
Intel and PGI/Nvidia work w/o problems on my Linux system.
We might try to solve it in a similar way.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (6 preceding siblings ...)
2020-12-04 18:37 ` anlauf at gcc dot gnu.org
@ 2020-12-04 21:32 ` anlauf at gcc dot gnu.org
2020-12-04 22:09 ` anlauf at gcc dot gnu.org
` (4 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: anlauf at gcc dot gnu.org @ 2020-12-04 21:32 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
--- Comment #8 from anlauf at gcc dot gnu.org ---
Created attachment 49687
--> https://gcc.gnu.org/bugzilla/attachment.cgi?id=49687&action=edit
Untested patch (proof of concept)
Here's a possible patch that retries after short reads.
Not regtested.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (7 preceding siblings ...)
2020-12-04 21:32 ` anlauf at gcc dot gnu.org
@ 2020-12-04 22:09 ` anlauf at gcc dot gnu.org
2020-12-04 22:28 ` tkoenig at gcc dot gnu.org
` (3 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: anlauf at gcc dot gnu.org @ 2020-12-04 22:09 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
anlauf at gcc dot gnu.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|WAITING |NEW
--- Comment #9 from anlauf at gcc dot gnu.org ---
The patch seems to regtest ok, but certainly needs some wider testing.
NIST? Other I/O test suites?
May also need some cosmetic cleanup to match GNU coding style.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (8 preceding siblings ...)
2020-12-04 22:09 ` anlauf at gcc dot gnu.org
@ 2020-12-04 22:28 ` tkoenig at gcc dot gnu.org
2020-12-04 22:38 ` anlauf at gcc dot gnu.org
` (2 subsequent siblings)
12 siblings, 0 replies; 14+ messages in thread
From: tkoenig at gcc dot gnu.org @ 2020-12-04 22:28 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
--- Comment #10 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
(In reply to anlauf from comment #9)
> The patch seems to regtest ok, but certainly needs some wider testing.
Actually, I think the bug is in io/unix.c:raw_read. That should take
care of repeating the reads as needed.
Seems like that, if nbyte <= MAX_CHUNK, we do not take account of the
possibility of a short read.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (9 preceding siblings ...)
2020-12-04 22:28 ` tkoenig at gcc dot gnu.org
@ 2020-12-04 22:38 ` anlauf at gcc dot gnu.org
2021-01-21 21:58 ` anlauf at gcc dot gnu.org
2021-01-22 12:12 ` anlauf at gcc dot gnu.org
12 siblings, 0 replies; 14+ messages in thread
From: anlauf at gcc dot gnu.org @ 2020-12-04 22:38 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
--- Comment #11 from anlauf at gcc dot gnu.org ---
(In reply to Thomas Koenig from comment #10)
> Seems like that, if nbyte <= MAX_CHUNK, we do not take account of the
> possibility of a short read.
Yes, that seems to be the better/right place.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (10 preceding siblings ...)
2020-12-04 22:38 ` anlauf at gcc dot gnu.org
@ 2021-01-21 21:58 ` anlauf at gcc dot gnu.org
2021-01-22 12:12 ` anlauf at gcc dot gnu.org
12 siblings, 0 replies; 14+ messages in thread
From: anlauf at gcc dot gnu.org @ 2021-01-21 21:58 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
anlauf at gcc dot gnu.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #49687|0 |1
is obsolete| |
--- Comment #12 from anlauf at gcc dot gnu.org ---
Created attachment 50025
--> https://gcc.gnu.org/bugzilla/attachment.cgi?id=50025&action=edit
Revised patch, addressing the issue in unix.c
Hi Thomas,
I've regtested the attached patch. It removes the first (problematic) part
of raw_read, and fixes the second part. It seems to regtest fine, BUT:
I had one (random?) failure with append_1.f90 which I am unable to reproduce.
Looking at the strace output, it appears that there is one additional read(2)
with the patch that returns 0. This is reproducible. I have not yet found
out why this happens, but then I am probably to blind-sighted.
But it fixes the testcase.
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Bug libfortran/98129] Failure on reading big chunk of /dev/urandom
2020-12-03 20:01 [Bug libfortran/98129] New: Failure on reading big chunk of /dev/urandom tkoenig at gcc dot gnu.org
` (11 preceding siblings ...)
2021-01-21 21:58 ` anlauf at gcc dot gnu.org
@ 2021-01-22 12:12 ` anlauf at gcc dot gnu.org
12 siblings, 0 replies; 14+ messages in thread
From: anlauf at gcc dot gnu.org @ 2021-01-22 12:12 UTC (permalink / raw)
To: gcc-bugs
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98129
--- Comment #13 from anlauf at gcc dot gnu.org ---
(In reply to anlauf from comment #12)
> I had one (random?) failure with append_1.f90 which I am unable to reproduce.
Likely origin: parallel make -j4 check-fortran and
% grep -il open.*[\"\']foo[\'\"] gcc/testsuite/gfortran.dg/ -r
gcc/testsuite/gfortran.dg/append_1.f90
gcc/testsuite/gfortran.dg/direct_io_2.f90
gcc/testsuite/gfortran.dg/io_constraints_7.f03
gcc/testsuite/gfortran.dg/large_unit_2.f90
gcc/testsuite/gfortran.dg/open_access_1.f90
gcc/testsuite/gfortran.dg/open_access_append_1.f90
gcc/testsuite/gfortran.dg/open_status_1.f90
gcc/testsuite/gfortran.dg/pr20163-2.f
The additional read seen in the strace output comes from the first read
attempt on the file which is shorter (169 bytes) that the buffer (8k),
so there is another read attempt (returning 0) because of end of file.
So that is a feature of the present fix.
^ permalink raw reply [flat|nested] 14+ messages in thread