[Bug fortran/50937] New: STAT option with ALLOCATE statement on large arrays

[Bug fortran/50937] New: STAT option with ALLOCATE statement on large arrays

[fwi at inducks dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

             Bug #: 50937
           Summary: STAT option with ALLOCATE statement on large arrays
    Classification: Unclassified
           Product: gcc
           Version: 4.4.3
            Status: UNCONFIRMED
          Severity: minor
          Priority: P3
         Component: fortran
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: fwi@inducks.org


I'm reporting something that looks like a bug, at least I can't imagine how
this behavior could be considered right, but I admit I haven't read the entire
C and Fortran specifications that gfortran is following.

When specifying the "STAT" option with the ALLOCATE statement, no error is
returned for some extremely large arrays that obviously can not fit into
memory.

test.f90:

  PROGRAM MAIN
  IMPLICIT NONE
  DOUBLE PRECISION, ALLOCATABLE :: E(:,:,:,:)
  INTEGER :: N, ierr
  CHARACTER(LEN=32) :: arg
  CALL get_command_argument(1, arg)
  READ(arg,*) N
  ALLOCATE(E(N,N,N,N),STAT=ierr)
  IF (ierr/=0) THEN
    PRINT *,'Could not allocate array of size ',N,'**4, error code=', ierr
  ELSE
    PRINT *,'Sucesfully allocated array of size ',N,'**4'
    E(N,N,N,N)=1.0D0
    DEALLOCATE(E)
  ENDIF
  END PROGRAM MAIN

Run with:

$ gfortran test.f90; for i in `seq -w 10 10 400`; do ./a.out $i; done
 Sucesfully allocated array of size           10 **4
 Sucesfully allocated array of size           20 **4
 Sucesfully allocated array of size           30 **4
 Sucesfully allocated array of size           40 **4
 Sucesfully allocated array of size           50 **4
 Sucesfully allocated array of size           60 **4
 Sucesfully allocated array of size           70 **4
 Sucesfully allocated array of size           80 **4
 Sucesfully allocated array of size           90 **4
 Sucesfully allocated array of size          100 **4
 Sucesfully allocated array of size          110 **4
 Sucesfully allocated array of size          120 **4
 Could not allocate array of size          130 **4, error code=        5014
 Could not allocate array of size          140 **4, error code=        5014
 Could not allocate array of size          150 **4, error code=        5014
 Sucesfully allocated array of size          160 **4
 Could not allocate array of size          170 **4, error code=        5014
 Could not allocate array of size          180 **4, error code=        5014
 Sucesfully allocated array of size          190 **4
 Could not allocate array of size          200 **4, error code=        5014
 Could not allocate array of size          210 **4, error code=        5014
 Sucesfully allocated array of size          220 **4
 Sucesfully allocated array of size          230 **4
 Sucesfully allocated array of size          240 **4
 Sucesfully allocated array of size          250 **4
 Could not allocate array of size          260 **4, error code=        5014


It's obvious that on my 3Gb-RAM machine, the 250**4 array was NOT allocated.

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[kargl at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

kargl at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kargl at gcc dot gnu.org

--- Comment #1 from kargl at gcc dot gnu.org 2011-10-31 18:06:32 UTC ---
What operating system and does it use a virtual memory
system?  Change 'e(n,n,n,n) = 1' to 'e = 1' and see 
what happens.  In one case, you are touching only the
last page in vm while in the other you are touching
all pages.

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[fwi at inducks dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #2 from fwi at inducks dot org 2011-10-31 18:15:57 UTC ---
With "E=1.0D0" instead of "E(N,N,N,N)=1.0D0"

$ gfortran test.f90; for i in `seq -w 10 10 400`; do LANG=C ./a.out $i; done
 Sucesfully allocated array of size           10 **4
 Sucesfully allocated array of size           20 **4
 Sucesfully allocated array of size           30 **4
 Sucesfully allocated array of size           40 **4
 Sucesfully allocated array of size           50 **4
 Sucesfully allocated array of size           60 **4
 Sucesfully allocated array of size           70 **4
 Sucesfully allocated array of size           80 **4
 Sucesfully allocated array of size           90 **4
 Sucesfully allocated array of size          100 **4
 Sucesfully allocated array of size          110 **4
 Sucesfully allocated array of size          120 **4
 Could not allocate array of size          130 **4, error code=        5014
 Could not allocate array of size          140 **4, error code=        5014
 Could not allocate array of size          150 **4, error code=        5014
 Sucesfully allocated array of size          160 **4
Erreur de segmentation
 Could not allocate array of size          170 **4, error code=        5014
 Could not allocate array of size          180 **4, error code=        5014
 Sucesfully allocated array of size          190 **4
Erreur de segmentation
 Could not allocate array of size          200 **4, error code=        5014
 Could not allocate array of size          210 **4, error code=        5014
 Sucesfully allocated array of size          220 **4
Erreur de segmentation
 Sucesfully allocated array of size          230 **4
Erreur de segmentation
 Sucesfully allocated array of size          240 **4
Erreur de segmentation
 Sucesfully allocated array of size          250 **4
Erreur de segmentation
 Could not allocate array of size          260 **4, error code=        5014
 Could not allocate array of size          270 **4, error code=        5014
 Sucesfully allocated array of size          280 **4
Erreur de segmentation
 Sucesfully allocated array of size          290 **4
Erreur de segmentation
 Sucesfully allocated array of size          300 **4
Erreur de segmentation
 Sucesfully allocated array of size          310 **4
Erreur de segmentation
 Could not allocate array of size          320 **4, error code=        5014
 Sucesfully allocated array of size          330 **4
Erreur de segmentation
 Could not allocate array of size          340 **4, error code=        5014
 Could not allocate array of size          350 **4, error code=        5014
 Sucesfully allocated array of size          360 **4
Erreur de segmentation
 Could not allocate array of size          370 **4, error code=        5014
 Could not allocate array of size          380 **4, error code=        5014
 Sucesfully allocated array of size          390 **4
Erreur de segmentation
 Could not allocate array of size          400 **4, error code=        5014

(Excuse the non-English words, "Erreur de segmentation" really means
"Segmentation fault").

I am using Ubuntu 10.04.3 LTS (32-bit) with:

$ cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : 
stepping    : 10
cpu MHz        : 2493.666
cache size    : 1024 KB
physical id    : 0
siblings    : 2
core id        : 0
cpu cores    : 2
apicid        : 0
initial apicid    : 0
fdiv_bug    : no
hlt_bug        : no
f00f_bug    : no
coma_bug    : no
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc
arch_perfmon pebs bts aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3
cx16 xtpr pdcm xsave lahf_lm tpr_shadow vnmi flexpriority
bogomips    : 4987.33
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #3 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-10-31 18:24:14 UTC ---
>From the metadata, it seems you're using version 4.4.3, where the overflow
check when calculating the size to allocate was a bit stupid. Basically it did
the calculation, and if the result was negative, overflow was detected. But
this check missed the cases where the calculated values would wrap around to a
positive value again. I believe this is what you're seeing when some of the
intermediate values succeed. In current versions of the compiler, overflow
check is bit smarter. See PR 28105.

Also, as Steve Kargl hinted at, many operating systems allow memory to be
overcommitted. I.e. allocating memory succeeds, but the program is killed later
on if it actually tries to use that memory. Gfortran has no checks whether the
amount of memory you try to allocate is some reasonable fraction of the memory
installed in the machine, it just does what you tell it to do.

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[fwi at inducks dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #4 from fwi at inducks dot org 2011-10-31 18:29:27 UTC ---
    I'm using:

    $ gfortran --version
    GNU Fortran (Ubuntu 4.4.3-4ubuntu5) 4.4.3

    I've now tested the same program on a 64-bit CentOs machine 
    with 16-Gb RAM, but wasn't able to reproduce the problem there:

    $ gfortran --version
    GNU Fortran (GCC) 4.1.2

    $ for i in `seq -w 10 10 400`; do LANG=C ./a.out $i; done
     Sucesfully allocated array of size           10 **4
     Sucesfully allocated array of size           20 **4
     Sucesfully allocated array of size           30 **4
     Sucesfully allocated array of size           40 **4
     Sucesfully allocated array of size           50 **4
     Sucesfully allocated array of size           60 **4
     Sucesfully allocated array of size           70 **4
     Sucesfully allocated array of size           80 **4
     Sucesfully allocated array of size           90 **4
     Sucesfully allocated array of size          100 **4
     Sucesfully allocated array of size          110 **4
     Sucesfully allocated array of size          120 **4
     Sucesfully allocated array of size          130 **4
     Sucesfully allocated array of size          140 **4
     Sucesfully allocated array of size          150 **4
     Sucesfully allocated array of size          160 **4
     Sucesfully allocated array of size          170 **4
     Sucesfully allocated array of size          180 **4
     Sucesfully allocated array of size          190 **4
     Sucesfully allocated array of size          200 **4
     Sucesfully allocated array of size          210 **4
     Could not allocate array of size          220 **4, error code=           1
     Could not allocate array of size          230 **4, error code=           1
     Could not allocate array of size          240 **4, error code=           1
     Could not allocate array of size          250 **4, error code=           1
     Could not allocate array of size          260 **4, error code=           1
     Could not allocate array of size          270 **4, error code=           1
     Could not allocate array of size          280 **4, error code=           1
     Could not allocate array of size          290 **4, error code=           1
     Could not allocate array of size          300 **4, error code=           1
     Could not allocate array of size          310 **4, error code=           1
     Could not allocate array of size          320 **4, error code=           1
     Could not allocate array of size          330 **4, error code=           1
     Could not allocate array of size          340 **4, error code=           1
     Could not allocate array of size          350 **4, error code=           1
     Could not allocate array of size          360 **4, error code=           1
     Could not allocate array of size          370 **4, error code=           1
     Could not allocate array of size          380 **4, error code=           1
     Could not allocate array of size          390 **4, error code=           1
     Could not allocate array of size          400 **4, error code=           1

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

Janne Blomqvist <jb at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
                 CC|                            |jb at gcc dot gnu.org
         Resolution|                            |INVALID

--- Comment #5 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-10-31 19:01:59 UTC ---
(In reply to comment #4)
>     I've now tested the same program on a 64-bit CentOs machine 
>     with 16-Gb RAM, but wasn't able to reproduce the problem there:

Yes, because on a 64-bit platform the size calculation is done with 64-bit
integers, and long before such integers overflow the OS decides to deny the
allocation request (as can be seen e.g. by the different error code).

I'm not really seeing any bug here.

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[fwi at inducks dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #6 from fwi at inducks dot org 2011-10-31 19:25:38 UTC ---
Has the bug been corrected in recent versions of gfortran, or do you really
mean it's OK that gfortran claims an array has been allocated when it really
has not been?

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[sgk at troutmask dot apl.washington.edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #7 from Steve Kargl <sgk at troutmask dot apl.washington.edu> 2011-10-31 19:50:41 UTC ---
On Mon, Oct 31, 2011 at 07:25:38PM +0000, fwi at inducks dot org wrote:
> 
> Has the bug been corrected in recent versions of gfortran, or do you really
> mean it's OK that gfortran claims an array has been allocated when it really
> has not been?
> 

Yes, the problem of integer overflow that Janne mentioned has
been corrected.

As to gfortran claiming "that an array has been allocated when
it really has not been", suggests that you may not understand
how your operating system works.  The concise story is that
gfortran asks your OS for a gazillion bytes of memory and
please confirm the status.  The OS tells gfortran (actually
your compiled code), here's your gazillion bytes of virtual
memory and everything is a-okay.  As long as you do not 
actually touch the pages of allocated virtual memory, no
physical memory has been allocated to your program.  Once
you start touching pages, at some point the OS notices a
shortage of physical memory.  This shortage leads to swapping
other (idle) processes out to secondary storage, utilizing
swap space as the backing storage for your virtual memory,
evidently your process exceeds its limits or the system
limits and the process segfaults.

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[fwi at inducks dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #8 from fwi at inducks dot org 2011-10-31 20:17:51 UTC ---
I do not(In reply to comment #7)
> On Mon, Oct 31, 2011 at 07:25:38PM +0000, fwi at inducks dot org wrote:
> Yes, the problem of integer overflow that Janne mentioned has
> been corrected.

Great.

I indeed do not know everything about the OS and what it does when I "allocate"
an array. But that's exactly the purpose of a programming language like
Fortran, an abstraction that should be "good enough" for programing without
having to know everything about the OS.
Secondly, users are sometimes better than programmers at telling them if
something is really useful or not. In that case, the question is: what is the
purpose of the STAT flag in an allocate STATEMENT if it won't give you any
reasonable indication if the array you have can be used or not.

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[sgk at troutmask dot apl.washington.edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #9 from Steve Kargl <sgk at troutmask dot apl.washington.edu> 2011-10-31 21:02:52 UTC ---
On Mon, Oct 31, 2011 at 08:17:51PM +0000, fwi at inducks dot org wrote:
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937
> 
> --- Comment #8 from fwi at inducks dot org 2011-10-31 20:17:51 UTC ---
> I do not(In reply to comment #7)
> > On Mon, Oct 31, 2011 at 07:25:38PM +0000, fwi at inducks dot org wrote:
> > Yes, the problem of integer overflow that Janne mentioned has
> > been corrected.
> 
> Great.
> 
> I indeed do not know everything about the OS and what it does when I "allocate"
> an array. But that's exactly the purpose of a programming language like
> Fortran, an abstraction that should be "good enough" for programing without
> having to know everything about the OS.
> Secondly, users are sometimes better than programmers at telling them if
> something is really useful or not. In that case, the question is: what is the
> purpose of the STAT flag in an allocate STATEMENT if it won't give you any
> reasonable indication if the array you have can be used or not.
> 

Use a newer version of gfortran 4.1.2 and 4.4.3 are old.  Install
4.6.2 and see what happens.  You've been told twice that the 
integer overflow has been fixed.

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #10 from Janne Blomqvist <jb at gcc dot gnu.org> 2011-11-01 07:55:06 UTC ---
(In reply to comment #8)
> I indeed do not know everything about the OS and what it does when I "allocate"
> an array. But that's exactly the purpose of a programming language like
> Fortran, an abstraction that should be "good enough" for programing without
> having to know everything about the OS.

Sometimes abstractions leak, unfortunately. There's really not anything
gfortran can do about that. And, it's not unique to gfortran either. gfortran
ALLOCATE uses the C malloc(), as does e.g. the default implementation of the
C++ new operator and presumably a lot of other language runtimes as well. So
they all share the same issues in case the system overcommits memory.

I can certainly sympathize with the notion that memory overcommit is inane and
shouldn't be enabled by default, but that's a system policy issue and nothing
gfortran can do anything about. As you're on Linux, FWIW you can disable
overcommit by setting the "vm.overcommit_memory" sysctl to the value 2. See
http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting

> Secondly, users are sometimes better than programmers at telling them if
> something is really useful or not. In that case, the question is: what is the
> purpose of the STAT flag in an allocate STATEMENT if it won't give you any
> reasonable indication if the array you have can be used or not.

Indeed, on a system which overcommits memory, ALLOCATE with STAT is not
particularly useful. But, again, memory overcommitting is a system policy issue
and gfortran can't do anything about it.

And, one might add, if all you're going to do with the STAT result is checking
whether it's nonzero and stopping the program in that case, you might as well
not bother because that's exactly what ALLOCATE without STAT already does.

[Bug fortran/50937] STAT option with ALLOCATE statement on large arrays

[fwi at inducks dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50937

--- Comment #11 from fwi at inducks dot org 2011-11-01 12:00:54 UTC ---
(In reply to comment #10)
> Sometimes abstractions leak, unfortunately. There's really not anything
> gfortran can do about that. And, it's not unique to gfortran either. gfortran
> ALLOCATE uses the C malloc(), as does e.g. the default implementation of the
> C++ new operator and presumably a lot of other language runtimes as well. So
> they all share the same issues in case the system overcommits memory.

I hate to continue this discussion, but I think the integer overflow problem,
which is what I was reporting, was a different issue. The fact that you solved
it in recent gfortran versions proves that there was something you could do
about that.

> I can certainly sympathize with the notion that memory overcommit is inane and
> shouldn't be enabled by default, but that's a system policy issue and nothing
> gfortran can do anything about. As you're on Linux, FWIW you can disable
> overcommit by setting the "vm.overcommit_memory" sysctl to the value 2. See
> http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting

Thanks for the info.

> And, one might add, if all you're going to do with the STAT result is checking
> whether it's nonzero and stopping the program in that case, you might as well
> not bother because that's exactly what ALLOCATE without STAT already does.

Not really, because then I can directly tell users how they can solve the
issue, that is either switch to a 64bit OS or compile the MPI version of the
same program (because then the main array is splitted in several chunks, all of
them small enough to be indexed with 32bit integers).
Without the STAT option, users will be left in the dark with just "it doesn't
work". Or with "I need to spend time reading the manual".

And please - I hope you all do not take my remarks as harsh criticism, I do
appreciate the efforts and job you did with gfortran. A few years ago, my code
was much faster with ifort than gcc's Fortran, nowadays it's comparable and I
can tell people to use gfortran if they like.