Help with long compile time of all-USE module

Help with long compile time of all-USE module

[Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

All,

I'm hoping to rely on the wisdom of the gfortran gurus for some help on this.

I've been looking at trying to speed up the compilation of a library I help maintain because, well, always a worthy goal, especially with CI and Git. I did some profiling and found that the most expensive file to compile in our library (doing a debug build with GCC 10.3) is a "meta-module" that only has 'use foo' statements:

https://github.com/GEOS-ESM/MAPL/blob/main/base/MAPL_Mod.F90

It allows us to "gather" many use-statements so users don't need to "use" 50 different modules.

This really surprised me as there are other files in this library that are infinitely more complex and I can think of ways we could maybe refactor or break them up, but this file is...boring. I guess I could split it up, but almost doesn't seem worth the effort.

Now, it is the slowest compile when building with Debugging flags which are, for this model:

  FLAGS = -O0 -g -fcheck=all,no-array-temps -finit-real=snan -ffree-line-length-none -fno-range-check -Wno-missing-include-dirs -fbacktrace -ffpe-trap=zero,overflow -fbacktrace -fallow-argument-mismatch -fallow-invalid-boz -falign-commons -Jinclude/MAPL.base -fPIC -ffixed-line-length-132 -pthread -fopenmp

My naïve thought when I first saw this was the tall pole was "well, I'll degrade the optimization" but, well, already at -O0. So perhaps one of these other flags is triggering some sort of weirdness with an "only USE" file?

I might try a binary exclusion experiment to figure it out (remove half the flags, etc.), but maybe it's obvious to the experts.

Thanks,
Matt

--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

Aw: Help with long compile time of all-USE module

[Martin Stein]

Hi Matthew,
 
please have a look at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98426 but ignore any patches I have attached there, because they are just wrong. It definitely sound like you hit the same problem. As described there, if you have a lot of symbols within the mod files pulled in by the use statements, then an O(N^2) search increases compilation time considerably.
 
Best regards Martin
 
 
 

Gesendet: Mittwoch, 12. Mai 2021 um 15:52 Uhr
Von: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via Fortran" <fortran@gcc.gnu.org>
An: "fortran@gcc.gnu.org" <fortran@gcc.gnu.org>
Betreff: Help with long compile time of all-USE module
All,

I'm hoping to rely on the wisdom of the gfortran gurus for some help on this.

I've been looking at trying to speed up the compilation of a library I help maintain because, well, always a worthy goal, especially with CI and Git. I did some profiling and found that the most expensive file to compile in our library (doing a debug build with GCC 10.3) is a "meta-module" that only has 'use foo' statements:

https://github.com/GEOS-ESM/MAPL/blob/main/base/MAPL_Mod.F90

It allows us to "gather" many use-statements so users don't need to "use" 50 different modules.

This really surprised me as there are other files in this library that are infinitely more complex and I can think of ways we could maybe refactor or break them up, but this file is...boring. I guess I could split it up, but almost doesn't seem worth the effort.

Now, it is the slowest compile when building with Debugging flags which are, for this model:

FLAGS = -O0 -g -fcheck=all,no-array-temps -finit-real=snan -ffree-line-length-none -fno-range-check -Wno-missing-include-dirs -fbacktrace -ffpe-trap=zero,overflow -fbacktrace -fallow-argument-mismatch -fallow-invalid-boz -falign-commons -Jinclude/MAPL.base -fPIC -ffixed-line-length-132 -pthread -fopenmp

My naïve thought when I first saw this was the tall pole was "well, I'll degrade the optimization" but, well, already at -O0. So perhaps one of these other flags is triggering some sort of weirdness with an "only USE" file?

I might try a binary exclusion experiment to figure it out (remove half the flags, etc.), but maybe it's obvious to the experts.

Thanks,
Matt

--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson[http://science.gsfc.nasa.gov/sed/bio/matthew.thompson]

Re: [EXTERNAL] Aw: Help with long compile time of all-USE module

[Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

Martin,

Thanks. I'll try to give it a try soon...

...though that might be a while. Our preliminary tests with GCC 11 seem to show some issues with our code (other than slowness). Our devs (including Tom Clune who might be known to some here) are looking to see why GCC 10.3 is happy, but GCC 11 isn't. The changelog didn't seem to exciting but you never know when things like polymorphism, strings, et al are all happening at once!

Matt

--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

From: Martin Stein <mscfd@gmx.net>
Date: Monday, May 17, 2021 at 6:15 AM
To: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson@nasa.gov>
Cc: "fortran@gcc.gnu.org" <fortran@gcc.gnu.org>
Subject: [EXTERNAL] Aw: Help with long compile time of all-USE module

PS: I am currently using (with 11.1) the following simple patch to significantly speeding up my compilation. The patch is not correct, but I could not find any examples, where this makes a difference. Even by looking at the orignal commit introducing the find_symbol call I could not come up with an example where this matters. I would be interested to know whether this is the cause for the long compilation times in your case as well.

gcc/fortran/module.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gcc/fortran/module.c b/gcc/fortran/module.c
index 089453caa03..a29865c7412 100644
--- a/gcc/fortran/module.c
+++ b/gcc/fortran/module.c
@@ -5501,11 +5501,12 @@ read_module (void)
            this symbol, which is not in an ONLY clause, must not be
            added to the namespace(11.3.2).  Note that find_symbol
            only returns the first occurrence that it finds.  */
-         if (!only_flag && !info->u.rsym.renamed
+/*       if (!only_flag && !info->u.rsym.renamed
               && strcmp (name, module_name) != 0
               && find_symbol (gfc_current_ns->sym_root, name,
                               module_name, 0))
           continue;
+*/

         st = gfc_find_symtree (gfc_current_ns->sym_root, p);


Best regards
Martin