[Bug fortran/96018] New: Optimization issue with external HDF5 library

[Bug fortran/96018] New: Optimization issue with external HDF5 library

[martin.schlipf at damnthespam dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

            Bug ID: 96018
           Summary: Optimization issue with external HDF5 library
           Product: gcc
           Version: 10.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: fortran
          Assignee: unassigned at gcc dot gnu.org
          Reporter: martin.schlipf at damnthespam dot com
  Target Milestone: ---

Created attachment 48817
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48817&action=edit
Minimal example to demonstrate the issue.

On a clean Ubuntu 20.04 with the following setup

sudo apt-get update
sudo apt-get install gfortran-10
sudo apt-get install libhdf5-dev

the attached code produces an error when compiled with optimization

gfortran-10 -O2 hdf5_bug.f90 -I/usr/include/hdf5/serial/
-L/usr/lib/x86_64-linux-gnu/ -lhdf5_serial_fortran

As far as I can tell, the reason this appears is that the compiler fails to
recognize that the function call in line 98 has side effects and optimizes it
away. If you check the status of the error code, the issue disappears.
Strangely, the exact same sequence of calls with a 2d array instead of a 3d one
works fine.

[Bug fortran/96018] Optimization issue with external HDF5 library

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2020-07-01

--- Comment #1 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
Can you please provide the file(s) needed for

use hdf5

? TIA.

[Bug fortran/96018] Optimization issue with external HDF5 library

[martin.schlipf at damnthespam dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #2 from martin.schlipf at damnthespam dot com ---
Well hdf5 is not developed by me, its a huge library. You can install it
manually if you want (https://www.hdfgroup.org/solutions/hdf5/), but it is
available on Ubuntu as mentioned

sudo apt-get install libhdf5-dev

There is probably a similar package for other linux distributions.

[Bug fortran/96018] Optimization issue with external HDF5 library

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #3 from anlauf at gcc dot gnu.org ---
You are not using HDF5's "native" Fortran interface directly, but a
clumsy way with c_f_pointer to obscure your code.  Any reason for that?

Have you considered using RESHAPE for what you seem to try to get?

Your code will run into issues if the dummy "array" is not contiguous.

Also, I'd consider not checking a library's return code bad style.

Also, can you give details on the exact compiler version and HDF5?

[Bug fortran/96018] Optimization issue with external HDF5 library

[martin.schlipf at damnthespam dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #4 from martin.schlipf at damnthespam dot com ---
Hdf5 doesn't have native support for complex datatypes, so we convert to real
and write that it is a complex as an attribute. If you replace the conversion
logic by an array instead of a pointer

allocate(real_array(2,size(array,1),size(array,2),size(array,3)))
reshape(transfer(array, real_array), shape(real_array))

it doesn't alert the behavior.

I agree that the error code should be checked and in fact, I already started a
merge request for that particular part. But for this bug report, I need to
leave it away, because otherwise the compiler does the right thing.

Finally, I reproduced it with gfortran 9.3.0 + hdf 1.12.0 and gfortran 10.0 +
hdf 1.10.4. With older versions of gfortran 7.3.0 it does not appear.

[Bug fortran/96018] Optimization issue with external HDF5 library

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #5 from anlauf at gcc dot gnu.org ---
(In reply to martin.schlipf from comment #4)
> Finally, I reproduced it with gfortran 9.3.0 + hdf 1.12.0 and gfortran 10.0
> + hdf 1.10.4. With older versions of gfortran 7.3.0 it does not appear.

Workarounds:

- compile with -O0, not sure why this makes a difference

- or replace in write_real_array_nd:

    real,             intent(in) :: array(*)

by

    real                         :: array(*)



I'd recommend to additionally replace in write_complex_array_3d

   ierr = write_real_array_4d(locid, dataset_name, ptr)

by

   ierr = write_real_array_4d(locid, dataset_name, &
                              reshape (transfer(array,[1.0]),[2,
shape(array)]))

The latter is IMO much cleaner Fortran.

There might by a bug in gfortran with the way you use c_f_pointer and
assumed-size, which generates wrong code for argument packing, or there
is a bug elsewhere which may corrupt data subtly.

[Bug fortran/96018] Optimization issue with external HDF5 library

[martin.schlipf at damnthespam dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #6 from martin.schlipf at damnthespam dot com ---
Sorry, if that has not been clear enough. I already know how to work around
this issue. You can simply check the error flag [if (ierr /= 0) return].

What I do not understand is why gfortran removes the call to the first function
when I switch on the optimization.

[Bug fortran/96018] Optimization issue with external HDF5 library

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #7 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
I can not test at the moment, that will have to wait for a few days.

A general comment:

In Fortran, functions exist to return a value. C-style „return an error status“
fit rather badly to the language, that is much better expressed wirh a
subroutine
(plus an optional error reporting argument).

Having said that, one thing that might influence the behavior is front-end
optimization. What happens if -fno-frontend-optimize -O is specified?

[Bug fortran/96018] Optimization issue with external HDF5 library

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #8 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
Comment on attachment 48817
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48817
Minimal example to demonstrate the issue.

Hm, I cannot reproduce this because I do not have the hdf5 library
installed.

Is there a way to create a self-contained test case?  I'd like to
run a bisection on this.

[Bug fortran/96018] Optimization issue with external HDF5 library

[martin.schlipf at damnthespam dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

martin.schlipf at damnthespam dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #48817|0                           |1
        is obsolete|                            |

--- Comment #9 from martin.schlipf at damnthespam dot com ---
Created attachment 48833
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48833&action=edit
Minimal example to demonstrate the issue.

The example is now independent of HDF5 which makes it much simpler.

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |9.4
           Priority|P3                          |P4
            Summary|Optimization issue with     |[9/10/11 Regression]
                   |external HDF5 library       |Optimization issue with
                   |                            |external HDF5 library
      Known to fail|                            |10.1.0, 11.0, 9.3.0
      Known to work|                            |8.4.1
             Status|WAITING                     |NEW
                 CC|                            |rguenth at gcc dot gnu.org

--- Comment #10 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
Confirmed, the change occurred between regions r262584 (2018-07-12, OK) and
r263045 (2018-07-28, wrong code). I am not sure this a gfortran bug).

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |needs-bisection

--- Comment #11 from Richard Biener <rguenth at gcc dot gnu.org> ---
So on trunk I get with -O2 (with or without -fno-inline)

> ./a.out 
write text
write text
 3d works
write text
 4d does not

and with -O0

> ./a.out 
write text
write text
 3d works
write text
write text
 4d does not

where write_4d is optimized to

write_4d ()
{
  <bb 2> [local count: 1073741824]:
  # .MEM_22 = VDEF <.MEM_1(D)>
  call_side_effect (); [tail call]
  # VUSE <.MEM_22>
  return;

}

the interesting fact is that inner_4d is computed "pure" (no side-effects)
while inner_3d is not and this is so from the start so done by the FE.

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #12 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
I don't have a debuggable source here at the moment, but I think there
may be a problem with implicit_pure, which was either introduced by
a patch in the range that Dominique provided (maybe for PR 85599?), or
exposed by something in the middle end.

The problem can be seen with -fdump-tree-original (in my case, with 9.3.0):

Namespace: A-Z: (UNKNOWN 0)
procedure name = wrapper
  symtree: 'call_side_effect'|| symbol: 'call_side_effect'
    type spec : (INTEGER 4)
    attributes: (PROCEDURE MODULE-PROC  FUNCTION)
    result: ierr
  symtree: 'inner_3d'    || symbol: 'inner_3d'
    type spec : (INTEGER 4)
    attributes: (PROCEDURE MODULE-PROC  FUNCTION)
    result: ierr
    Formal arglist: array
  symtree: 'inner_4d'    || symbol: 'inner_4d'
    type spec : (INTEGER 4)
    attributes: (PROCEDURE MODULE-PROC  FUNCTION IMPLICIT_PURE)
    result: ierr
    Formal arglist: array

IMPLICIT_PURE should not be set on inner_4d, because it evaluates the
function call_side_effect (and marking is as PURE is correctly rejected
because of that).

It is also strange that inner_3d is not also wrongly marked as
IMPLICIT_PURE, I do not really see a significant difference between

   integer function inner_3d(array) result(ierr)
      real, intent(in) :: array(:,:,:)
      integer dimensions(3)
      dimensions = shape(array)
      ierr = call_side_effect()
   end function inner_3d

and

   integer function inner_4d(array) result(ierr)
      real, intent(in) :: array(:,:,:,:)
      integer dimensions(4)
      dimensions = shape(array)
      ierr = call_side_effect()
   end function inner_4d

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #13 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
In the last comment I meant -fdump-fortran-original, of course.

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |tkoenig at gcc dot gnu.org

--- Comment #14 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
Created attachment 48852
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48852&action=edit
Patch which ought to work

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|needs-bisection             |patch

--- Comment #15 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
Patch at https://gcc.gnu.org/pipermail/gcc-patches/2020-July/549860.html .

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #16 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Thomas Kथà¤nig <tkoenig@gcc.gnu.org>:

https://gcc.gnu.org/g:3055d879edb1bc2a3923f92a5e681c8f6774fbc3

commit r11-2215-g3055d879edb1bc2a3923f92a5e681c8f6774fbc3
Author: Thomas Koenig <tkoenig@gcc.gnu.org>
Date:   Sun Jul 19 12:23:43 2020 +0200

    Fix handling of implicit_pure by checking if non-pure procedures are
called.

    Procedures are marked as implicit_pure if they fulfill the criteria of
    pure procedures.  In this case, a procedure was not marked as not being
    implicit_pure which called another procedure, which had not yet been
    marked as not being implicit_impure.

    Fixed by iterating over all procedures, setting callers of procedures
    which are non-pure and non-implicit_pure as non-implicit_pure and
    doing this until no more procedure has been changed.

    gcc/fortran/ChangeLog:

    2020-07-19  Thomas Koenig  <tkoenig@gcc.gnu.org>

            PR fortran/96018
            * frontend-passes.c (gfc_check_externals): Adjust formatting.
            (implicit_pure_call): New function.
            (implicit_pure_expr): New function.
            (gfc_fix_implicit_pure): New function.
            * gfortran.h (gfc_fix_implicit_pure): New prototype.
            * parse.c (translate_all_program_units): Call
gfc_fix_implicit_pure.

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #17 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-10 branch has been updated by Thomas Kथà¤nig
<tkoenig@gcc.gnu.org>:

https://gcc.gnu.org/g:3373a73172d0e39fe054ab2d4bbff746b36c0fe8

commit r10-8520-g3373a73172d0e39fe054ab2d4bbff746b36c0fe8
Author: Thomas Koenig <tkoenig@gcc.gnu.org>
Date:   Thu Jul 23 17:56:09 2020 +0200

    Fix handling of implicit_pure by checking if non-pure procedures are
called.

    Procedures are marked as implicit_pure if they fulfill the criteria of
    pure procedures.  In this case, a procedure was not marked as not being
    implicit_pure which called another procedure, which had not yet been
    marked as not being implicit_impure.

    Fixed by iterating over all procedures, setting callers of procedures
    which are non-pure and non-implicit_pure as non-implicit_pure and
    doing this until no more procedure has been changed.

    Backport from trunk r11-2215-g3055d879edb1bc2a3923f92a5e681c8f6774fbc3 .

    gcc/fortran/ChangeLog:

    2020-07-10  Thomas Koenig  <tkoenig@gcc.gnu.org>

            PR fortran/96018
            * frontend-passes.c (gfc_check_externals): Adjust formatting.
            (implicit_pure_call): New function.
            (implicit_pure_expr): New function.
            (gfc_fix_implicit_pure): New function.
            * gfortran.h (gfc_fix_implicit_pure): New prototype.
            * parse.c (translate_all_program_units): Call
gfc_fix_implicit_pure.

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

--- Comment #18 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-9 branch has been updated by Thomas Kथà¤nig
<tkoenig@gcc.gnu.org>:

https://gcc.gnu.org/g:7be5add5534f4310b294f1b7ad8855a424edb8c1

commit r9-8761-g7be5add5534f4310b294f1b7ad8855a424edb8c1
Author: Thomas Koenig <tkoenig@gcc.gnu.org>
Date:   Thu Jul 23 20:26:10 2020 +0200

    Fix handling of implicit_pure by checking if non-pure procedures are
called.

    Procedures are marked as implicit_pure if they fulfill the criteria of
    pure procedures.  In this case, a procedure was not marked as not being
    implicit_pure which called another procedure, which had not yet been
    marked as not being implicit_impure.

    Fixed by iterating over all procedures, setting callers of procedures
    which are non-pure and non-implicit_pure as non-implicit_pure and
    doing this until no more procedure has been changed.

    Backport from trunk r11-2215-g3055d879edb1bc2a3923f92a5e681c8f6774fbc3 .

    gcc/fortran/ChangeLog:

    2020-07-10  Thomas Koenig  <tkoenig@gcc.gnu.org>

            PR fortran/96018
            * frontend-passes.c (gfc_check_externals): Adjust formatting.
            (implicit_pure_call): New function.
            (implicit_pure_expr): New function.
            (gfc_fix_implicit_pure): New function.
            * gfortran.h (gfc_fix_implicit_pure): New prototype.
            * parse.c (translate_all_program_units): Call
gfc_fix_implicit_pure.

[Bug fortran/96018] [9/10/11 Regression] Optimization issue with external HDF5 library

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96018

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #19 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
Fixed on all affected branches, closing.

Thanks a lot for the bug report!