[Bug fortran/41977] New: gfortran -fopenp and ACML_MP seem incompatible

[Bug fortran/41977] New: gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

gfortran -fopenmp is incompatible with -lacml_mp under at least some
circumstances.  The program is LAPACK Cholesky and its solver, converted
to Fortran 90 and instrumented for a course.  It fails with 4.4.1 in ACML
(i.e. the LAPACK call returns an erroneous error value), and in 4.3.2
the solver (which does NOT call ACML) takes 10 times as long as it should
do.  Omitting the -fopenmp and using -lacml instead both work perfectly;
i.e. it fails ONLY if both -fopenmp and -lacml_mp are used.  Oh, and it
works with -llapack, too.  Oh, joy.

Here is some grobble:

gosset$gfortran -v
Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../configure --prefix=/home/nmm/gfortran --disable-shared
--disable-threads --disable-bootstrap -enable-languages=fortran
--enable-werror=yes --enable-checking=all --disable-decimal-float
Thread model: single
gcc version 4.4.1 (GCC)
gosset$/usr/bin/gfortran -v
Using built-in specs.
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info
--mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64
--enable-languages=c,c++,objc,fortran,obj-c++,java,ada
--enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3
--enable-ssp --disable-libssp --with-bugurl=http://bugs.opensuse.org/
--with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap
--with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit
--enable-libstdcxx-allocator=new --disable-libstdcxx-pch
--enable-version-specific-runtime-libs --program-suffix=-4.3
--enable-linux-futex --without-system-libunwind --with-cpu=generic
--build=x86_64-suse-linux
Thread model: posix
gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux)

gfortran -Wall -std=f2003 -pedantic -O3 -fopenmp -o Cholesky_f \
    Answers/Cholesky/one.f90 -lacml_mp
./Cholesky_one_f fred_f_1000

The source is fairly short and clean, but fred_f_1000 is 16 MB.
The 4.4.1 failure shows with a 160 MB test file, but not the 4.3.2
one.


Regards,
Nick Maclaren,
University of Cambridge Computing Service,
New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
Email:  nmm1@cam.ac.uk
Tel.:  +44 1223 334761    Fax:  +44 1223 334679


-- 
           Summary: gfortran -fopenp and ACML_MP seem incompatible
           Product: gcc
           Version: unknown
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: fortran
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: nmm1 at cam dot ac dot uk


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #1 from nmm1 at cam dot ac dot uk  2009-11-07 17:44 -------
Created an attachment (id=18988)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=18988&action=view)
The source of the failing program


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #2 from nmm1 at cam dot ac dot uk  2009-11-07 17:46 -------
Created an attachment (id=18989)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=18989&action=view)
The first half of the data generation code


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #3 from nmm1 at cam dot ac dot uk  2009-11-07 17:46 -------
Created an attachment (id=18990)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=18990&action=view)
The second half of the generation code


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #4 from nmm1 at cam dot ac dot uk  2009-11-07 17:48 -------
Sorry - there's no need to send you the data file - I can send the programs
I use to generate it, which are very short.  I have done so.  To run these,
compile them into binaries of the same name with almost any options, and
run them by:

    Generate 1000 fred


-- 

nmm1 at cam dot ac dot uk changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |nmm1 at cam dot ac dot uk


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[kargl at gcc dot gnu dot org]

------- Comment #5 from kargl at gcc dot gnu dot org  2009-11-07 18:24 -------
Nick,

Have you also reported this issue to the ACML forum?
AFAICT, acml is a closed source library (at least I
couldn't find the source code), and the problem may 
in fact be within acml.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #6 from nmm1 at cam dot ac dot uk  2009-11-07 18:29 -------
Subject: Re:  gfortran -fopenp and ACML_MP seem incompatible

No, but I will try to find it. It is definitely a closed source library, 
and was written by NAG for AMD under contract. I quite agree that the bug 
could be anywhere - in particular, from its symptoms, it definitely nowhere 
near whey symptoms are appearing :-(

I suspect a symbol clash, run-time incompatibility or some such.

Regards,
Nick.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[rguenth at gcc dot gnu dot org]

------- Comment #7 from rguenth at gcc dot gnu dot org  2009-11-07 18:40 -------
I suspect acml_mp is built using a different GCC version and they link
libgomp statically.  Can you check what the acml_mp library links against?
Just use ldd for this.  Then check that this is consistent with what your
test program uses.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #8 from nmm1 at cam dot ac dot uk  2009-11-07 18:56 -------
Subject: Re:  gfortran -fopenp and ACML_MP seem incompatible

On Nov 7 2009, rguenth at gcc dot gnu dot org wrote:
>
> ------- Comment #7 from rguenth at gcc dot gnu dot org 2009-11-07 18:40 
> ------- I suspect acml_mp is built using a different GCC version and they 
> link libgomp statically. Can you check what the acml_mp library links 
> against? Just use ldd for this. Then check that this is consistent with 
> what your test program uses.

Nothing noticeable.  The only lines that are in the executable and not
acml_mp are:
      libacml_mp.so => /usr/local/lib64/libacml_mp.so
      libgcc_s.so.1 => /lib64/libgcc_s.so.1
and the only line that is in acml_mp but not acml is:
      libgomp.so.1 => /usr/lib64/libgomp.so.1

No other differences (checked by diff), except for the hex. grobble in
brackets.  Checking the binary for the lapack version shows that the only
difference from the failing one is the ACML libraries are replaced by
LAPACK and BLAS.

It could still be a gcc version problem, of course, but not in that respect.

Regards,
Nick.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[kargl at gcc dot gnu dot org]

------- Comment #9 from kargl at gcc dot gnu dot org  2009-11-07 19:00 -------
Even without acml, there appears to be an issue.

gfortran44 -o one -O2 -pipe -march=native one.f90 -L/usr/local/lib \
           -llapack -lblas -fopenmp
./one fred_f_1000
LAPACK Cholesky time = .27 seconds, CPU = .27 seconds
LAPACK solver time = 2.48 seconds, CPU = 2.47 seconds
Coded Cholesky time = .81 seconds, CPU = 1.47 seconds
Coded solver time = 33.76 seconds, CPU = 38.68 seconds

gfortran44 -o one -O2 -pipe -march=native one.f90 \
           -L/usr/local/lib -llapack -lblas 
./one fred_f_1000
LAPACK Cholesky time = .30 seconds, CPU = .30 seconds
LAPACK solver time = 2.49 seconds, CPU = 2.49 seconds
Coded Cholesky time = 1.36 seconds, CPU = 1.36 seconds
Coded solver time = 2.97 seconds, CPU = 2.96 seconds

OpenMP clearly helps the 'Coded Cholesky time', but it
causes a factor of 10 degradation in the 'Coded solver time'.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #10 from nmm1 at cam dot ac dot uk  2009-11-07 19:15 -------
Subject: Re:  gfortran -fopenp and ACML_MP seem incompatible

On Nov 7 2009, kargl at gcc dot gnu dot org wrote:
>
>OpenMP clearly helps the 'Coded Cholesky time', but it
>causes a factor of 10 degradation in the 'Coded solver time'.

Ah!  Now, that's what I saw for acml_mp on 4.3.2, reliably, but
never saw it for the others.  It's clearly the sort of problem that
appears and disappears.  Oh, how I do love those :-(

Regards,
Nick Maclaren.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[burnus at gcc dot gnu dot org]

------- Comment #11 from burnus at gcc dot gnu dot org  2009-11-07 22:19 -------
Which version of ACML are you using? I get a bit confused with all those
4-dot-something version numbers as both GCC and ACML have them.

Do you use ACML 4.3.0?

 strings /opt/acml4.3.0/gfortran64/lib/libacml.a |grep GCC |sort -u
 strings /opt/acml4.3.0/gfortran64_mp/lib/libacml_mp.so |grep GCC |sort -u
shows here:
 Built using Fortran compiler: GNU Fortran (GCC) 4.3.2

Thus it should be ABI compatible with GCC 4.3.x, 4.4.x and 4.5.x.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #12 from nmm1 at cam dot ac dot uk  2009-11-07 23:33 -------
Subject: Re:  gfortran -fopenp and ACML_MP seem incompatible

On Nov 7 2009, burnus at gcc dot gnu dot org wrote:
>
> ------- Comment #11 from burnus at gcc dot gnu dot org 2009-11-07 22:19 
> ------- Which version of ACML are you using? I get a bit confused with 
> all those 4-dot-something version numbers as both GCC and ACML have them.
>
>Do you use ACML 4.3.0?

Yes.  I am pretty sure that it's something more obscure.  While I could
make guesses, none of them would have more than a few percent chance of
being right!

Regards,
Nick Maclaren.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #13 from nmm1 at cam dot ac dot uk  2009-11-10 13:32 -------
Oh, joy.  This went soft yesterday, and today I can't repeat the effects.
I have some of the evidence in Email I sent to the ACML expert, who has
managed to repeat them at least once.  But, at BEST, the effects seem to
depend on static/dynamic linking, Clib versions, Old Uncle Tom Cobbley and
all.  Here is the main effect I was reporting:

gosset$gfortran -Wall -std=f2003 -pedantic -O3 -fopenmp -o Cholesky_one_f \
gosset>    Answers/Cholesky/one.f90 -lacml_mp
gosset$./Cholesky_one_f fred_f_1000
 Error in calling LAPACK

Instrumenting the source showed that was INFO=15, which indicates that
pivot 15 was zero, which is ridiculous.  And it is THAT one which was the
bug that primarily concerned me.


I may have made some mistakes in identifying the performance issues.  Given
where the OpenMP directives are in the solver, some slowdown is to be expected.
That is clearly not a bug.  I distinctly remember having to break in after
4+ minutes using gfortran 4.3.2, when the 'normal' time was about 30 seconds.
That could be a Linux scheduler problem, of course.

Until and unless I can produce some more demonstrably incorrect performance,
I suggest ignoring that aspect of my report.  If I can't repeat it, either
I got myself confused, or it will be too evil to justify the effort of
investigating.  In either case ....


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[jvdelisle at gcc dot gnu dot org]

------- Comment #14 from jvdelisle at gcc dot gnu dot org  2009-11-10 20:30 -------
Marking status as waiting.


-- 

jvdelisle at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenp and ACML_MP seem incompatible

[nmm1 at cam dot ac dot uk]

------- Comment #15 from nmm1 at cam dot ac dot uk  2009-11-10 21:49 -------
It's come back again.  The more that it does that, the more that I am convinced
that it is something horrible in thread management, perhaps a symbol clash,
race
condition or overwriting, and the root cause might well not be in gfortran or
ACML.  Ugh.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977

[Bug fortran/41977] gfortran -fopenmp and ACML_MP seem incompatible

[dfranke at gcc dot gnu dot org]

------- Comment #16 from dfranke at gcc dot gnu dot org  2010-05-07 16:41 -------
Any news here? Can this one be closed?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41977