[Bug libfortran/48587] New: Avoid exhausting unit number with NEWUNIT=

[Bug libfortran/48587] New: Avoid exhausting unit number with NEWUNIT=

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48587

           Summary: Avoid exhausting unit number with NEWUNIT=
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: enhancement
          Priority: P3
         Component: libfortran
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: jb@gcc.gnu.org


As was recently pointed out by Tobias Burnus in a thread on comp.lang.fortran,
the current implementation of NEWUNIT= doesn't reuse unit numbers. Hence it's
possible that a program might exhaust the available unit numbers (this requires
that the program repeatedly closes and reopens files, as OS's have limits on
the number of file descriptors a process may have concurrently opened,
typically 1024 or something like that)

Currently there is just a (mutex protected) static variable which is
decremented for each time an OPEN statement with NEWUNIT= is issued, with a
wraparound check that generates an error if wraparound is detected.

IMHO an elegant solution would be to just reuse the kernel provided file
descriptor. E.g. 

int fd = open(...);
if (fd == -1)
{
    /* Handle error... */
}
new_unit_number = -fd;

This should work because a successful open() will always return a positive fd
number (see
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html ), and
the kernel takes care of reusing file descriptor numbers of closed files.

[Bug libfortran/48587] Avoid exhausting unit number with NEWUNIT=

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48587

Tobias Burnus <burnus at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |burnus at gcc dot gnu.org

--- Comment #1 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-04-13 10:20:00 UTC ---
There is a problem with reusing numbers - it is probably only a small one - but
it might be larger than the problem of running out of unit IDs:

The user might reuse NEWUNIT= IDs:

Open(newunit=id)
...
close(id)
open(unit=id) ...

The latter is illegal as one may only pass positive numbers to unit=, but I
would not count on it. Given that the number is huge, we decided that the
current method of just incrementing the value should be sufficient for nearly
all cases and one can think about something clever if a real-space code
encounters the problem. However, I don't mind if one solves the problem before.

[Bug libfortran/48587] Avoid exhausting unit number with NEWUNIT=

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48587

--- Comment #2 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-04-13 10:46:42 UTC ---
(In reply to comment #1)
> There is a problem with reusing numbers - it is probably only a small one - but
> it might be larger than the problem of running out of unit IDs:
> 
> The user might reuse NEWUNIT= IDs:
> 
> Open(newunit=id)
> ...
> close(id)
> open(unit=id) ...
> 
> The latter is illegal as one may only pass positive numbers to unit=, but I
> would not count on it.

I don't see how this would differ from the current behavior vs. my suggested
implementation. In any case, we check that the UNIT= number is positive when
opening units:

program negative_unit
  implicit none
  open(-10, file="foo.txt")
  write(-10, *) "Hello world"
  close(-10)
end program negative_unit

$ ./negative_unit 
At line 3 of file negative_unit.f90
Fortran runtime error: Bad unit number in OPEN statement

(this is with 4.4; presumably this restriction hasn't been lifted?)

Both the current implementation and my suggested one ensure that any unit
number returned in NEWUNIT= is negative. This ensures that the UNIT= and
NEWUNIT= number ranges are kept separate from each other, so they shouldn't
interact.

> Given that the number is huge, we decided that the
> current method of just incrementing the value should be sufficient for nearly
> all cases and one can think about something clever if a real-space code
> encounters the problem.

Ah, but what about a k-space code? ;)

> However, I don't mind if one solves the problem before.

Yeah, well, I don't think this is a big deal; after all it's pretty unlikely
that any program will overflow the counter.

[Bug libfortran/48587] Avoid exhausting unit number with NEWUNIT=

[jvdelisle at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48587

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jvdelisle at gcc dot
                   |                            |gnu.org

--- Comment #3 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> 2011-04-13 18:58:47 UTC ---
If someone overflows the counter I think they should get what they deserve.  ;)

I suppose one could put an OPEN .. CLOSE in a DO LOOP and see what happens now.
It might take a while to get there. IIRC when I implemented this I set the
limit small and tested an error check for this, but maybe I am wrong.

[Bug libfortran/48587] Avoid exhausting unit number with NEWUNIT=

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48587

--- Comment #4 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-04-14 08:12:33 UTC ---
(In reply to comment #3)
> I suppose one could put an OPEN .. CLOSE in a DO LOOP and see what happens now.

With gfortran 4.4.3 on Linux 2.6.32, /tmp on ext4, single SATA disk, and 2.67
GHz Xeon X3450 CPU, the following program

program twobfiles
  implicit none
  integer, parameter :: dp = kind(1.0d0)
  integer :: ii, cnt, cnt2, cnt_rate
  real(dp) :: elapsed
  call system_clock(cnt, cnt_rate)
  do ii = 10, 10010
     open(unit=ii, status="scratch")
     close(ii)
  end do
  call system_clock(cnt2)
  elapsed = real((cnt2 - cnt), dp) / cnt_rate
  print *, "Estimate for opening and closing 2 billion files: ", &
       elapsed * 2._dp**31 / 10000, " s."
end program twobfiles

estimates around 10 hours. I suspect current trunk and changing the code to use
NEWUNIT= are in the same ballpark.

[Bug libfortran/48587] Avoid exhausting unit number with NEWUNIT=

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48587

--- Comment #5 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-04-15 09:13:14 UTC ---
Reading the standard, my simple suggestion in #1 will not actually work. One
reason is that the NEWUNIT value may not be -1; with my approach this is
possible if one first closes fd 1. Secondly, it's possible to reassociate also
a negative unit number with a new file; but there is not guarantee that the
kernel will assign the same file descriptor to the new file. Wrt. I found a bug
in the current implementation, see PR 48618.

[Bug libfortran/48587] Avoid exhausting unit number with NEWUNIT=

[fxcoudert at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48587

Francois-Xavier Coudert <fxcoudert at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2011.04.19 13:40:08
                 CC|                            |fxcoudert at gcc dot
                   |                            |gnu.org
     Ever Confirmed|0                           |1

--- Comment #6 from Francois-Xavier Coudert <fxcoudert at gcc dot gnu.org> 2011-04-19 13:40:08 UTC ---
(In reply to comment #4)
> estimates around 10 hours. I suspect current trunk and changing the code to use
> NEWUNIT= are in the same ballpark.

With a tmpfs filesystem, I get that down to 1.5 hour. Might be worth fixing at
some point.

[Bug libfortran/48587] Avoid exhausting unit number with NEWUNIT=

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48587

Janne Blomqvist <jb at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at gcc dot gnu.org      |jb at gcc dot gnu.org

--- Comment #7 from Janne Blomqvist <jb at gcc dot gnu.org> ---
I have a preliminary patch, which I'm planning to finish and submit when 4.10
stage 1 opens. Assigning to myself.