[Bug fortran/49540] New: [4.6/4.7 Regression] Memory-hot with large DATA stmt

[Bug fortran/49540] New: [4.6/4.7 Regression] Memory-hot with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

           Summary: [4.6/4.7 Regression] Memory-hot with large DATA stmt
           Product: gcc
           Version: 4.6.1
            Status: UNCONFIRMED
          Keywords: memory-hog
          Severity: normal
          Priority: P3
         Component: fortran
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: jakub@gcc.gnu.org


COMMON/A/B(100000,100)
      DATA B/10000000*0.0/
      END
used to compile in just a couple of MB of memory and fraction of a second, but
starting with
http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=159076
it needs over 5GB of RAM and very long time.
That is only small fragment from original real-world testcase, which contains:
      PARAMETER (MAXSTR=150001)
c...
        COMMON/HJJET2/NSG,NJSG(MAXSTR),IASG(MAXSTR,3),K1SG(MAXSTR,100),
     &       K2SG(MAXSTR,100),PXSG(MAXSTR,100),PYSG(MAXSTR,100),
     &       PZSG(MAXSTR,100),PESG(MAXSTR,100),PMSG(MAXSTR,100)
c...
        DATA NSG/0/,NJSG/150001*0/,IASG/450003*0/,
     &       K1SG/15000100*0/,K2SG/15000100*0/,
     &       PXSG/15000100*0.0/,PYSG/15000100*0.0/,PZSG/15000100*0.0/,
     &       PESG/15000100*0.0/,PMSG/15000100*0.0/
c...
      END
c...
which compiled just fine with gcc up to 4.5.x and with 4.6/4.7 it is basically
out of any hope to compile it.

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to work|                            |4.5.2
            Summary|[4.6/4.7 Regression]        |[4.6/4.7 Regression]
                   |Memory-hot with large DATA  |Memory-hog with large DATA
                   |stmt                        |stmt
      Known to fail|                            |4.6.0, 4.7.0

--- Comment #1 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-06-27 08:34:10 UTC ---
Even if the initializer isn't just all zeros, 4.5.x compiles it very fast and
with minimum amount of memory.

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |4.6.1

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

Tobias Burnus <burnus at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |burnus at gcc dot gnu.org,
                   |                            |dfranke at gcc dot gnu.org,
                   |                            |jvdelisle at gcc dot
                   |                            |gnu.org

--- Comment #2 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-06-27 12:31:53 UTC ---
The patch in question (Rev. 159076, cf. link in comment 0) is:

2010-05-05  Daniel Franke  <franke.daniel@gmail.com>
    PR fortran/24978
    * gfortran.h: Removed repeat count from constructor, removed
    all usages.

That PR fixed an ICE on invalid code - and ensured gfortran diagnoses code
like:
  real :: e(3)
  data       e    / 3*1 /
  data       e(2) / 2 /
where "e"'s second element is multiple times initialized (which is not
diagnosed with -std=gnu, only with -std=f2008 - with gnu/legacy and with ifort,
the second initialization is plainly ignored).

Thus, in some way, the repeat count must come back. One possibility is to
handle the special case /<array_size>*<value>/, which is equivalent to a scalar
initialization. I think that should cover the most common case.

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

--- Comment #4 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-06-27 12:34:58 UTC ---
See also submitted patches at
  http://gcc.gnu.org/ml/fortran/2010-04/msg00328.html
  http://gcc.gnu.org/ml/fortran/2010-05/msg00005.html

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|4.6.1                       |4.6.2

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-06-27 12:33:07 UTC ---
GCC 4.6.1 is being released.

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

--- Comment #5 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-06-27 15:28:31 UTC ---
(In reply to comment #2)
> Thus, in some way, the repeat count must come back. One possibility is to
> handle the special case /<array_size>*<value>/, which is equivalent to a scalar
> initialization. I think that should cover the most common case.

Vaguely related to the latter: PR 49278, which is about mixing (default)
initialization of DT with DATA initialization.

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

--- Comment #7 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-06-27 17:22:15 UTC ---
(In reply to comment #6)
> (In reply to comment #2)
> > Thus, in some way, the repeat count must come back. One possibility is to
> > handle the special case /<array_size>*<value>/, which is equivalent to
> > a scalar initialization. I think that should cover the most common case.
> 
> IIRC, as implemented the array constructor is completely unrolled, for each
> element an entry in the splay tree is generated.

Yes - at least since dropping the repeat count, which your patch did.

> One could apply the -fmax-array-constructor=<value> restrictions used e.g. for
> PARAMETER arrays and do not unroll arrays of size larger than <value> but do it
> on runtime.

As written, I think it should be sufficient to support the initialization via a
scalar. In terms of the trans*.c files that already works:
  real :: a(3) = 0
thus, there is no reason why one should be able to set sym->value also for
  real a(3)
  data a/3*0/

That is: If - and only if - one has <repeat count> == size(array), i.e. a
whole-array initialization, one changes the initialization from:
  sym->value  =  [ ( <value>, i = 1, <repeat count>) ]
to
  sym->value  =  <value>

That should require some reshoveling in the code, but sounds much cleaner as
-fmax-array-constructor= tweaking. Additionally, it will generate much faster
code for "data a/10000000*0.0/" as it gets translated into "static real a =
{}".

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[adrian.sevcenco at cern dot ch]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

Adrian Sevcenco <adrian.sevcenco at cern dot ch> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |adrian.sevcenco at cern dot
                   |                            |ch

--- Comment #8 from Adrian Sevcenco <adrian.sevcenco at cern dot ch> 2011-06-28 08:59:38 UTC ---
(In reply to comment #7)
> (In reply to comment #6)
> > (In reply to comment #2)
> > > Thus, in some way, the repeat count must come back. One possibility is to
> > > handle the special case /<array_size>*<value>/, which is equivalent to
> > > a scalar initialization. I think that should cover the most common case.
> > 
> > IIRC, as implemented the array constructor is completely unrolled, for each
> > element an entry in the splay tree is generated.
> 
> Yes - at least since dropping the repeat count, which your patch did.
> 
> > One could apply the -fmax-array-constructor=<value> restrictions used e.g. for
> > PARAMETER arrays and do not unroll arrays of size larger than <value> but do it
> > on runtime.
> 
> As written, I think it should be sufficient to support the initialization via a
> scalar. In terms of the trans*.c files that already works:
>   real :: a(3) = 0
> thus, there is no reason why one should be able to set sym->value also for
>   real a(3)
>   data a/3*0/
> 
> That is: If - and only if - one has <repeat count> == size(array), i.e. a
> whole-array initialization, one changes the initialization from:
>   sym->value  =  [ ( <value>, i = 1, <repeat count>) ]
> to
>   sym->value  =  <value>
> 
> That should require some reshoveling in the code, but sounds much cleaner as
> -fmax-array-constructor= tweaking. Additionally, it will generate much faster
> code for "data a/10000000*0.0/" as it gets translated into "static real a =
> {}".

so with regard to the initial submission from 
https://bugzilla.redhat.com/show_bug.cgi?id=716721
(and code from
http://alisoft.cern.ch/viewvc/trunk/TAmpt/AMPT/hijing1.383_ampt.f?view=markup&root=AliRoot)

do you have some specific (and certain) recommendations in order to make this
scientific code to be compiled (alongside with a lot other fortran scientific
code)?
Thanks

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

--- Comment #9 from Tobias Burnus <burnus at gcc dot gnu.org> 2011-06-28 12:20:41 UTC ---
(In reply to comment #8)
> do you have some specific (and certain) recommendations in order to make this
> scientific code to be compiled (alongside with a lot other fortran scientific
> code)?

As short term solution: Use an older version of gfortran (before
4.6.0-2010-05-05) - or remove the 0 initialization. While the Fortran standard
does not guarantee it, in practice, static memory should nevertheless be zero
initialized (.bss).

The proper fix will take a bit longer: A couple of days for the fix and then it
will likely also take a while until a Red Hat/Fedora build will be available.
(Though, non-Fedora nightly builds will have it earlier.)

 * * *

(In reply to comment #7)
> As written, I think it should be sufficient to support the initialization
> via a scalar.

For what it is worth: Intel's compiler seem to do the same. Hence,
  DATA B/<repeat>*<value>/
is is very fast while using
  DATA B(:)/<repeat>*<value>/
is very slow - even though, both lines are effectively the same.

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |ASSIGNED
   Last reconfirmed|                            |2011.06.29 16:54:35
         AssignedTo|unassigned at gcc dot       |jakub at gcc dot gnu.org
                   |gnu.org                     |
     Ever Confirmed|0                           |1

--- Comment #10 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-06-29 16:54:35 UTC ---
Created attachment 24634
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=24634
gcc47-pr49540.patch

Untested fix, which reintroduces repeat field.

[Bug fortran/49540] [4.6/4.7 Regression] Memory-hog with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

--- Comment #11 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-06-30 10:25:43 UTC ---
Author: jakub
Date: Thu Jun 30 10:25:40 2011
New Revision: 175693

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=175693
Log:
    PR fortran/49540
    * gfortran.h (gfc_constructor): Add repeat field.
    * trans-array.c (gfc_conv_array_initializer): Handle repeat > 1.
    * array.c (current_expand): Add repeat field.
    (expand_constructor): Copy repeat.
    * constructor.c (node_free, node_copy, gfc_constructor_get,
    gfc_constructor_lookup): Handle repeat field.
    (gfc_constructor_lookup_next, gfc_constructor_remove): New functions.
    * data.h (gfc_assign_data_value): Add mpz_t * argument.
    (gfc_assign_data_value_range): Removed.
    * constructor.h (gfc_constructor_advance): Removed.
    (gfc_constructor_lookup_next, gfc_constructor_remove): New prototypes.
    * data.c (gfc_assign_data_value): Add REPEAT argument, handle it and
    also handle overwriting a range with a single entry.
    (gfc_assign_data_value_range): Removed.
    * resolve.c (check_data_variable): Adjust gfc_assign_data_value
    call.  Use gfc_assign_data_value instead of
    gfc_assign_data_value_expr.

    * gfortran.dg/pr49540-1.f90: New test.
    * gfortran.dg/pr49540-2.f90: New test.

Added:
    trunk/gcc/testsuite/gfortran.dg/pr49540-1.f90
    trunk/gcc/testsuite/gfortran.dg/pr49540-2.f90
Modified:
    trunk/gcc/fortran/ChangeLog
    trunk/gcc/fortran/array.c
    trunk/gcc/fortran/constructor.c
    trunk/gcc/fortran/constructor.h
    trunk/gcc/fortran/data.c
    trunk/gcc/fortran/data.h
    trunk/gcc/fortran/gfortran.h
    trunk/gcc/fortran/resolve.c
    trunk/gcc/fortran/trans-array.c
    trunk/gcc/testsuite/ChangeLog

[Bug fortran/49540] [4.6 Regression] Memory-hog with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to work|                            |4.7.0
            Summary|[4.6/4.7 Regression]        |[4.6 Regression] Memory-hog
                   |Memory-hog with large DATA  |with large DATA stmt
                   |stmt                        |
      Known to fail|4.7.0                       |

--- Comment #12 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-06-30 13:55:02 UTC ---
Fixed on the trunk, will backport to 4.6 soon.

[Bug fortran/49540] [4.6 Regression] Memory-hog with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

--- Comment #13 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-07-04 21:08:00 UTC ---
Author: jakub
Date: Mon Jul  4 21:07:57 2011
New Revision: 175827

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=175827
Log:
    Backported from mainline
    2011-06-06  Jakub Jelinek  <jakub@redhat.com>

    PR debug/49262
    * dwarf2out.c (native_encode_initializer): Decrement count in each
    iteration.

    2011-06-30  Jakub Jelinek  <jakub@redhat.com>

    PR fortran/49540
    * gfortran.h (gfc_constructor): Add repeat field.
    * trans-array.c (gfc_conv_array_initializer): Handle repeat > 1.
    * array.c (current_expand): Add repeat field.
    (expand_constructor): Copy repeat.
    * constructor.c (node_free, node_copy, gfc_constructor_get,
    gfc_constructor_lookup): Handle repeat field.
    (gfc_constructor_lookup_next, gfc_constructor_remove): New functions.
    * data.h (gfc_assign_data_value): Add mpz_t * argument.
    (gfc_assign_data_value_range): Removed.
    * constructor.h (gfc_constructor_advance): Removed.
    (gfc_constructor_lookup_next, gfc_constructor_remove): New prototypes.
    * data.c (gfc_assign_data_value): Add REPEAT argument, handle it and
    also handle overwriting a range with a single entry.
    (gfc_assign_data_value_range): Removed.
    * resolve.c (check_data_variable): Adjust gfc_assign_data_value
    call.  Use gfc_assign_data_value instead of
    gfc_assign_data_value_expr.

    * gfortran.dg/pr49540-1.f90: New test.
    * gfortran.dg/pr49540-2.f90: New test.

Added:
    branches/gcc-4_6-branch/gcc/testsuite/gfortran.dg/pr49540-1.f90
    branches/gcc-4_6-branch/gcc/testsuite/gfortran.dg/pr49540-2.f90
Modified:
    branches/gcc-4_6-branch/gcc/ChangeLog
    branches/gcc-4_6-branch/gcc/dwarf2out.c
    branches/gcc-4_6-branch/gcc/fortran/ChangeLog
    branches/gcc-4_6-branch/gcc/fortran/array.c
    branches/gcc-4_6-branch/gcc/fortran/constructor.c
    branches/gcc-4_6-branch/gcc/fortran/constructor.h
    branches/gcc-4_6-branch/gcc/fortran/data.c
    branches/gcc-4_6-branch/gcc/fortran/data.h
    branches/gcc-4_6-branch/gcc/fortran/gfortran.h
    branches/gcc-4_6-branch/gcc/fortran/resolve.c
    branches/gcc-4_6-branch/gcc/fortran/trans-array.c
    branches/gcc-4_6-branch/gcc/testsuite/ChangeLog

[Bug fortran/49540] [4.6 Regression] Memory-hog with large DATA stmt

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49540

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

--- Comment #14 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-07-04 21:17:01 UTC ---
Fixed.