[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285
Bug 30285 depends on bug 25708, which changed state.

Bug 25708 Summary: [F95] Module loading is not good at all
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25708

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |FIXED

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[fxcoudert at gcc dot gnu dot org]

------- Comment #25 from fxcoudert at gcc dot gnu dot org  2007-11-17 13:49 -------
Fixed.


-- 

fxcoudert at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[fxcoudert at gcc dot gnu dot org]

------- Comment #24 from fxcoudert at gcc dot gnu dot org  2007-11-17 13:47 -------
Subject: Bug 30285

Author: fxcoudert
Date: Sat Nov 17 13:46:53 2007
New Revision: 130257

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=130257
Log:
        PR fortran/30285
        * module.c (struct written_common, written_commons): New structure.
        (compare_written_commons, free_written_common, write_common_0):
        New functions.
        (write_common): Call recursive function write_common_0.

Modified:
    trunk/gcc/fortran/ChangeLog
    trunk/gcc/fortran/module.c


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[patchapp at dberlin dot org]

------- Comment #23 from patchapp at dberlin dot org  2007-11-12 05:55 -------
Subject: Bug number PR 30285

A patch for this bug has been added to the patch tracker.
The mailing list url for the patch is
http://gcc.gnu.org/ml/gcc-patches/2007-11/msg00552.html


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[anlauf at gmx dot de]

------- Comment #22 from anlauf at gmx dot de  2007-11-11 22:18 -------
FX,

> I made a build of the patched compiler that you can download from
> http://www.coudert.name/tmp/gfortran-i686-linux-20071110.tar.gz

I successfully recompiled the app with OpenMPI enabled, which uses
several COMMON blocks.  Not only are the module files about a factor
100 smaller for me ;-), the compilation actually completes and the
executable works.  I am currently investigating the output, but the
problem described in this PR is solved for me.

Many thanks,
-ha


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[fxcoudert at gcc dot gnu dot org]

------- Comment #21 from fxcoudert at gcc dot gnu dot org  2007-11-10 18:09 -------
(In reply to comment #20)
> I would love to test it with a i586/i686-compatible build.  (f951 should
> be sufficient that can be used as a replacement in one of those regular
> builds provided on FX's web page.  Unfortunately I don't have the
> resourced to build gfortran myself.)

I made a build of the patched compiler that you can download from
http://www.coudert.name/tmp/gfortran-i686-linux-20071110.tar.gz

> I think the
> first "benchmark" is to have the module files of the reworked
> example to all have the same size and essentially same contents.
> From comment #14 I presume that this will indeed be the case.

Yes, it is the case. Please let us know how your real-life testing goes.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[anlauf at gmx dot de]

------- Comment #20 from anlauf at gmx dot de  2007-11-10 15:17 -------
Tobias,

> Harald, can you test the patch at
>   http://gcc.gnu.org/ml/gcc-patches/2007-11/msg00552.html
> with your real-world program?

I would love to test it with a i586/i686-compatible build.  (f951 should
be sufficient that can be used as a replacement in one of those regular
builds provided on FX's web page.  Unfortunately I don't have the
resourced to build gfortran myself.)

> And for the other tests I have, the compile time does not
> significantly change (too much noise to see the effect).

The essential problem was actually the virtual memory used by f951
when I last tried to compile the app with MPI enabled.  I think the
first "benchmark" is to have the module files of the reworked
example to all have the same size and essentially same contents.
>From comment #14 I presume that this will indeed be the case.

Thanks,
-ha


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[burnus at gcc dot gnu dot org]

------- Comment #19 from burnus at gcc dot gnu dot org  2007-11-10 12:29 -------
Harald, can you test the patch at
  http://gcc.gnu.org/ml/gcc-patches/2007-11/msg00552.html
with your real-world program? Due to .mod changes, the "gfc-excessive.tar.gz"
does not work. And for the other tests I have, the compile time does not
significantly change (too much noise to see the effect).


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[patchapp at dberlin dot org]

------- Comment #18 from patchapp at dberlin dot org  2007-11-10 04:01 -------
Subject: Bug number PR 30285

A patch for this bug has been added to the patch tracker.
The mailing list url for the patch is
http://gcc.gnu.org/ml/gcc-patches/2007-11/msg00552.html


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[Tobias dot Schlueter at physik dot uni-muenchen dot de]

------- Comment #17 from Tobias dot Schlueter at physik dot uni-muenchen dot de  2007-11-10 01:16 -------
Subject: Re:  gfortran excessive memory usage with COMMON
 blocks in modules

fxcoudert at gcc dot gnu dot org wrote:
> ------- Comment #16 from fxcoudert at gcc dot gnu dot org  2007-11-09 23:59 -------
> (In reply to comment #15)
>> I wrote this code originally, and I agree with your analysis.
> 
> But regtesting doesn't agree with my analysis... in case of common with
> bind(c,name="..."), this patch hampers the diagnosis of commons with the same
> name but different labels. I've tried hard to get it rolling that way, because
> I agree it's cleaner, but I couldn't. I'll propose a different approach (a
> hack, to avoid writing twice the same combination of name and binding label) to
> the list when it finishes regtesting.

There was no BIND(C) when I wrote it :-)

I'll give it a look tomorrow.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[fxcoudert at gcc dot gnu dot org]

------- Comment #16 from fxcoudert at gcc dot gnu dot org  2007-11-09 23:59 -------
(In reply to comment #15)
> I wrote this code originally, and I agree with your analysis.

But regtesting doesn't agree with my analysis... in case of common with
bind(c,name="..."), this patch hampers the diagnosis of commons with the same
name but different labels. I've tried hard to get it rolling that way, because
I agree it's cleaner, but I couldn't. I'll propose a different approach (a
hack, to avoid writing twice the same combination of name and binding label) to
the list when it finishes regtesting.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[Tobias dot Schlueter at physik dot uni-muenchen dot de]

------- Comment #15 from Tobias dot Schlueter at physik dot uni-muenchen dot de  2007-11-09 12:44 -------
Subject: Re:  gfortran excessive memory usage with COMMON
 blocks in modules

fxcoudert at gcc dot gnu dot org wrote:
> Because in that case, mod2.mod already has two copies of the common:
> 
>   (('mpipriv' 2 0 0 'mpipriv') ('mpipriv' 2 0 0 'mpipriv'))
> 
> and I don't think that's desirable. I think that the module loading is actually
> wrong here: the code in gfc_get_common (match.c) takes special care to
> duplicate this common name by creating a unique name for it. While I believe
> that mangling is necessary, the mangled name shouldn't be unique but simply
> prefixed, so that of the same name are merged, while prevented to clash with
> the namespace of the use'ing procedure.

I wrote this code originally, and I agree with your analysis.

> Index: match.c
> ===================================================================
> --- match.c     (revision 129869)
> +++ match.c     (working copy)
> @@ -2608,23 +2608,19 @@ gfc_common_head *
>  gfc_get_common (const char *name, int from_module)
>  {
>    gfc_symtree *st;
> -  static int serial = 0;
> -  char mangled_name[GFC_MAX_SYMBOL_LEN + 1];
> +  char mangled_name[GFC_MAX_SYMBOL_LEN + 12];

Should be + 13 (need one char for '\0').

> +    /* A use associated common block is only needed to correctly layout
> +       the variables it contains.  */
> +    snprintf (mangled_name, GFC_MAX_SYMBOL_LEN, "_frommodule_%s", name);

GFC_MAX_SYMBOL_LEN + 12, otherwise you could create ambiguities with 
really long common names.  Previously this wasn't possible due to the 
serial number.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[fxcoudert at gcc dot gnu dot org]

------- Comment #14 from fxcoudert at gcc dot gnu dot org  2007-11-09 12:36 -------
I think the problem starts with:

module mod0
  integer mpi_bottom
  common /mpipriv/ mpi_bottom
end module mod0

module mod1
  use mod0
end module mod1

module mod2
  use mod0
  use mod1
end module mod2


Because in that case, mod2.mod already has two copies of the common:

  (('mpipriv' 2 0 0 'mpipriv') ('mpipriv' 2 0 0 'mpipriv'))

and I don't think that's desirable. I think that the module loading is actually
wrong here: the code in gfc_get_common (match.c) takes special care to
duplicate this common name by creating a unique name for it. While I believe
that mangling is necessary, the mangled name shouldn't be unique but simply
prefixed, so that of the same name are merged, while prevented to clash with
the namespace of the use'ing procedure.

I'm regtesting the following patch:

Index: match.c
===================================================================
--- match.c     (revision 129869)
+++ match.c     (working copy)
@@ -2608,23 +2608,19 @@ gfc_common_head *
 gfc_get_common (const char *name, int from_module)
 {
   gfc_symtree *st;
-  static int serial = 0;
-  char mangled_name[GFC_MAX_SYMBOL_LEN + 1];
+  char mangled_name[GFC_MAX_SYMBOL_LEN + 12];

   if (from_module)
-    {
-      /* A use associated common block is only needed to correctly layout
-        the variables it contains.  */
-      snprintf (mangled_name, GFC_MAX_SYMBOL_LEN, "_%d_%s", serial++, name);
-      st = gfc_new_symtree (&gfc_current_ns->common_root, mangled_name);
-    }
-  else
-    {
-      st = gfc_find_symtree (gfc_current_ns->common_root, name);
-
-      if (st == NULL)
-       st = gfc_new_symtree (&gfc_current_ns->common_root, name);
-    }
+    /* A use associated common block is only needed to correctly layout
+       the variables it contains.  */
+    snprintf (mangled_name, GFC_MAX_SYMBOL_LEN, "_frommodule_%s", name);
+
+  st = gfc_find_symtree (gfc_current_ns->common_root,
+                        from_module ? mangled_name : name);
+
+  if (st == NULL)
+    st = gfc_new_symtree (&gfc_current_ns->common_root,
+                         from_module ? mangled_name : name);

   if (st->n.common == NULL)
     {


-- 

fxcoudert at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fxcoudert at gcc dot gnu dot
                   |                            |org
         AssignedTo|unassigned at gcc dot gnu   |fxcoudert at gcc dot gnu dot
                   |dot org                     |org
             Status|NEW                         |ASSIGNED
   GCC host triplet|i686-pc-linux-gnu           |
   Last reconfirmed|2006-12-24 10:47:43         |2007-11-09 12:36:15
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285

[Bug fortran/30285] gfortran excessive memory usage with COMMON blocks in modules

[anlauf at gmx dot de]

------- Comment #13 from anlauf at gmx dot de  2007-10-22 07:35 -------
I've adjusted the title slightly to refer to the use of
COMMON blocks (in MPICH, OpenMPI) because the problem of
.mod size excursion depends only on the "nesting depth",
not on the actual size of the modules.  I hope this bug
will be fixed someday when somebody has a look either at
the module writing routines or at the way the structures
containing module information are held internally.

I am unable to compile a reasonably large project (just in
terms of the number of files, not total source code size!)
using MPI.
The problem appears even worse with OpenMPI than with MPICH.


-- 

anlauf at gmx dot de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|gfortran excessive memory   |gfortran excessive memory
                   |usage with large modules    |usage with COMMON blocks in
                   |                            |modules


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30285