[Bug fortran/56008] New: [F03] lhs-allocation invoking the array-constructor on DDTs causes memory error

[Bug fortran/56008] New: [F03] lhs-allocation invoking the array-constructor on DDTs causes memory error

[stefan.mauerberger at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

             Bug #: 56008
           Summary: [F03] lhs-allocation invoking the array-constructor on
                    DDTs causes memory error
    Classification: Unclassified
           Product: gcc
           Version: 4.7.2
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: fortran
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: stefan.mauerberger@gmail.com


Created attachment 29183
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29183
minimal testcase

Hi There! 

I encountered some wired behavior with F03's automatic (re)allocation features
in 4.7.2 and 4.8.0 from trunk. 
What I have is a DDT with an allocatable member-variable. 
> TYPE :: test_typ
>    REAL, ALLOCATABLE :: a(:)
> END TYPE
I am initializing two instances of that type 'xx' and 'yy' with allocated
member-variables. In addition there is an allocatable array 'conc' of
'test_typ' which is not yet allocated. 
> TYPE(test_typ) :: xx, yy
> TYPE(test_typ), ALLOCATABLE :: conc(:)
> xx = test_typ( [1.0] )
> yy = test_typ( [1.0,2.0,3.0] )
To allocate 'conc' I want to utilize the array-constructor:
> conc = [ xx, yy ] 
This however, referring to Valgrinds memory analysis, causes a memory error: 
==14558== Conditional jump or move depends on uninitialised value(s)
==14558==    at 0x401030: MAIN__ (test.f90:16)
==14558==    by 0x40118B: main (test.f90:20)
==14558==  Uninitialised value was created by a heap allocation
==14558==    at 0x4C2B3F8: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14558==    by 0x400DCB: MAIN__ (test.f90:16)
==14558==    by 0x40118B: main (test.f90:20)

Well, 'gfortran -v -Wall -Wextra test.f90' compiles just fine without any
complaints. Also the executable produces results as expected (in most cases). 

The above problem, however, will be exposed and triggered by initializing the
MPI execution environment. Just invoking CALL MPI_INIT() will cause a
segmentation fault. 

First of all I wanted to blame OMPI for that (see:
http://www.open-mpi.org/community/lists/users/2013/01/21103.php). However,
those guys convinced me asking you for advice. 

To reproduce all that, there is a minimal test-case attached. In addition,
there is a file containing the system configuration and one with Valgrind's
memory analysis. 

Cheers, 
Stefan 

(Btw. what is the appropriate Fortran expression/terminology for DDT
member-variables)

[Bug fortran/56008] [F03] lhs-allocation invoking the array-constructor on DDTs causes memory error

[stefan.mauerberger at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

--- Comment #1 from stefan.mauerberger at gmail dot com 2013-01-16 19:37:56 UTC ---
Created attachment 29184
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29184
gfortran -v -Wall -Wextra -g test.f90

[Bug fortran/56008] [F03] lhs-allocation invoking the array-constructor on DDTs causes memory error

[stefan.mauerberger at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

--- Comment #2 from stefan.mauerberger at gmail dot com 2013-01-16 19:38:45 UTC ---
Created attachment 29185
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29185
valgrind -v --track-origins=yes ./a.out

[Bug fortran/56008] [F03] wrong code with lhs-realloc on assignment with derived types having allocatable components

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

Tobias Burnus <burnus at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
           Keywords|                            |wrong-code
   Last reconfirmed|                            |2013-01-16
                 CC|                            |burnus at gcc dot gnu.org,
                   |                            |pault at gcc dot gnu.org
     Ever Confirmed|0                           |1
            Summary|[F03] lhs-allocation        |[F03] wrong code with
                   |invoking the                |lhs-realloc on assignment
                   |array-constructor on DDTs   |with derived types having
                   |causes memory error         |allocatable components

--- Comment #3 from Tobias Burnus <burnus at gcc dot gnu.org> 2013-01-16 22:50:40 UTC ---
Seems to be similar to PR 47517.


(In reply to comment #0)
> (Btw. what is the appropriate Fortran expression/terminology for DDT
> member-variables)

"components". (Besides data and procedure-pointer components, a derived type
can also contain type-bound procedures.)


(In reply to comment #2)
> Created attachment 29185 [details]
> valgrind -v --track-origins=yes ./a.out

Two side remarks:

a) GCC 4.8 ships with -fsanitize=address,thread - and "address" ("ASAN")
roughly matches valgrind. Cf. http://gcc.gnu.org/gcc-4.8/changes.html - Though,
for that example, it doesn't seem to work. On the other hand, for stack-memory
issues, valgrind doesn't work and while ASAN might.

b) To track what goes wrong, one can also look at a C-like dump of the internal
representation using -fdump-tree-original (-fdump-tree-all, ...)


For:
   conc = [ xx, yy ]
one gets the following -ftree-dump-original:

                conc.data = (void * restrict) __builtin_malloc (96);
...
                    if ((void *) (*(struct test_typ[2] * restrict)
atmp.6.data)[S.9].a.data != 0B)


Either we have to use calloc instead of malloc - or we have to modify the
intrinsic assignment to assume unallocated memory in this case. In terms of
performance, the latter would be better. Using "calloc" is the quicker fix.



Side note 2: If one uses the GLIBC (i.e. Linux) and has MALLOC_PERTURB_ set,*
one gets a segfault when one tries to run the code. [MALLOC_PERTURB_ sets the
value returned by "malloc" to non-NUL.] (* e.g. "export
MALLOC_PERTURB_=$(($RANDOM % 255 + 1))")

[Bug fortran/56008] [F03] wrong code with lhs-realloc on assignment with derived types having allocatable components

[pault at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

Paul Thomas <pault at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|unassigned at gcc dot       |pault at gcc dot gnu.org
                   |gnu.org                     |

--- Comment #4 from Paul Thomas <pault at gcc dot gnu.org> 2013-01-19 15:54:50 UTC ---
Created attachment 29221
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=29221
Fix for this PR and PR 47517

Bootstraps and regtests OK

The data pointers of allocatable components need to be nullified, after
(re)allocation, before the assignment starts because of the test in allocation
for the allocation status of these components.

It's nearly 'obvious' but took me an age to get it :-)

I'll submit properly once I am back in France (tomorrow night if snow allows).

Paul

[Bug fortran/56008] [F03] wrong code with lhs-realloc on assignment with derived types having allocatable components

[dominiq at lps dot ens.fr]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

--- Comment #5 from Dominique d'Humieres <dominiq at lps dot ens.fr> 2013-01-20 13:34:52 UTC ---
> Created attachment 29221 [details]
> Fix for this PR and PR 47517

I confirm for this PR. However while the original code of PR 47517 executes
without error it has a huge memory leak:

==98070== HEAP SUMMARY:
==98070==     in use at exit: 146,401,340 bytes in 2,700,054 blocks
==98070==   total heap usage: 2,900,145 allocs, 200,091 frees, 170,502,858
bytes allocated
==98070== 
==98070== 12 bytes in 1 blocks are definitely lost in loss record 1 of 42
==98070==    at 0x100019679: malloc (vg_replace_malloc.c:266)
==98070==    by 0x100000EBB: MAIN__ (in ./a.out)
==98070==    by 0x100007BC9: main (in ./a.out)
...

==98070== 
==98070== 102,798,972 (71,999,280 direct, 30,799,692 indirect) bytes in 299,997
blocks are definitely lost in loss record 42 of 42
==98070==    at 0x100019679: malloc (vg_replace_malloc.c:266)
==98070==    by 0x1000072A7: MAIN__ (in ./a.out)
==98070==    by 0x100007BC9: main (in ./a.out)
==98070== 
==98070== LEAK SUMMARY:
==98070==    definitely lost: 105,601,152 bytes in 500,046 blocks
==98070==    indirectly lost: 40,800,100 bytes in 2,200,007 blocks
==98070==      possibly lost: 0 bytes in 0 blocks
==98070==    still reachable: 0 bytes in 0 blocks
==98070==         suppressed: 88 bytes in 1 blocks
==98070== 
==98070== For counts of detected and suppressed errors, rerun with: -v
==98070== ERROR SUMMARY: 34 errors from 34 contexts (suppressed: 0 from 0)

(loop reduded to 100000).

The test in PR 47517 comment #1 still fails at runtime:

 1>            1                    0
 2>            1                    0           1

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Compiling it with  -fsanitize=address gives

==237== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x000103be06b0
at pc 0x100000b75 bp 0x7fff5fbfd3a0 sp 0x7fff5fbfd398
READ of size 8 at 0x000103be06b0 thread T0

[Bug fortran/56008] [F03] wrong code with lhs-realloc on assignment with derived types having allocatable components

[pault at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

--- Comment #6 from Paul Thomas <pault at gcc dot gnu.org> 2013-01-21 19:30:21 UTC ---
(In reply to comment #5)
> > Created attachment 29221 [details]
> > Fix for this PR and PR 47517
> 
> I confirm for this PR. However while the original code of PR 47517 executes
> without error it has a huge memory leak:
> 

Indeed - I see it too and know exactly what the trouble is.

Please note that the ICE with the example in comment #1 is due to a
"deliberate" error by yours truly.... look at the mismatch between the scalar
references and the size of the allocated arrays.

***sigh***

Paul

PS I'm on to it!

[Bug fortran/56008] [F03] wrong code with lhs-realloc on assignment with derived types having allocatable components

[stefan.mauerberger at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

--- Comment #7 from stefan.mauerberger at gmail dot com 2013-01-22 19:08:10 UTC ---
Unfortunately, I do not understand a thing about all the internals and the
actual implementations. I just wanted to let you know that I am totally
overwhelmed how series bugs are treated and how quick bug-fixes approach. You
guys are just awesome! 
In case I can support you in any kind fixing that bug, please let me know.

[Bug fortran/56008] [F03] wrong code with lhs-realloc on assignment with derived types having allocatable components

[pault at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

--- Comment #8 from Paul Thomas <pault at gcc dot gnu.org> 2013-02-04 22:33:19 UTC ---
Author: pault
Date: Mon Feb  4 22:33:15 2013
New Revision: 195741

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=195741
Log:
2013-02-04  Paul Thomas  <pault@gcc.gnu.org>

    PR fortran/56008
    PR fortran/47517
    * trans-array.c (gfc_alloc_allocatable_for_assignment): Save
    the lhs descriptor before it is modified for reallocation. Use
    it to deallocate allocatable components in the reallocation
    block.  Nullify allocatable components for newly (re)allocated
    arrays.

2013-02-04  Paul Thomas  <pault@gcc.gnu.org>

    PR fortran/56008
    * gfortran.dg/realloc_on _assign_16.f90 : New test.

    PR fortran/47517
    * gfortran.dg/realloc_on _assign_17.f90 : New test.


Added:
    trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_16.f90
    trunk/gcc/testsuite/gfortran.dg/realloc_on_assign_17.f90
Modified:
    trunk/gcc/fortran/ChangeLog
    trunk/gcc/fortran/trans-array.c
    trunk/gcc/testsuite/ChangeLog

[Bug fortran/56008] [F03] wrong code with lhs-realloc on assignment with derived types having allocatable components

[pault at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

Paul Thomas <pault at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #9 from Paul Thomas <pault at gcc dot gnu.org> 2013-02-04 22:36:37 UTC ---
Fixed on trunk.  Thanks for the report, Stefan.

Paul

[Bug fortran/56008] [F03] wrong code with lhs-realloc on assignment with derived types having allocatable components

[stefan.mauerberger at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=56008

--- Comment #10 from stefan.mauerberger at gmail dot com 2013-02-10 18:34:51 UTC ---
Hi There!

Its me again. Unfortunately, I have to report that there is still something
wired. The following slightly modified code causes a memory leak: 
PROGRAM main

    TYPE :: test_typ
        REAL, ALLOCATABLE :: a(:)
    END TYPE 

    TYPE(test_typ) :: conc(2)

    conc = [ test_typ( [1.0,2.0] ), test_typ( [4.0,4.9] ) ]

END PROGRAM main

Well, the code does not crash, however, valgrind says:
...
==18920== LEAK SUMMARY:
==18920==    definitely lost: 16 bytes in 2 blocks
...

Not invoking the array-constructor, everything seems to be fine. Any ideas?