[Bug fortran/107317] New: [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[Bug fortran/107317] New: [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[gscfq@t-online.de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

            Bug ID: 107317
           Summary: [10/11/12/13 Regression] ICE in emit_redzone_byte, at
                    asan.cc:1508
           Product: gcc
           Version: 13.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: fortran
          Assignee: unassigned at gcc dot gnu.org
          Reporter: gscfq@t-online.de
  Target Milestone: ---

Started with r10 between 20190512 and 20190519,
with option -m32 and -fopenmp or -frecursive :


$ cat z1.f90
program p
   real :: a(1000000, 100, 2)
   call s(a(:, 1:200, :))
end


$ gfortran-13-20221016 -c z1.f90 -m32 -O2 -fsanitize=address -frecursive
z1.f90:3:15:

    3 |    call s(a(:, 1:200, :))
      |               1
Warning: Upper array reference at (1) is out of bounds (200 > 100) in dimension
2
z1.f90:1:9:

    1 | program p
      |         ^
Error: total size of local objects 2400000512 exceeds maximum 2147483392
during RTL pass: expand
z1.f90:1:9: internal compiler error: in emit_redzone_byte, at asan.cc:1508
0xf5c70e asan_redzone_buffer::emit_redzone_byte(long, unsigned char)
        ../../gcc/asan.cc:1508
0xf64786 asan_emit_stack_protection(rtx_def*, rtx_def*, unsigned int, long*,
tree_node**, int)
        ../../gcc/asan.cc:2003
0x9d9ba2 expand_used_vars
        ../../gcc/cfgexpand.cc:2376
0x9de182 execute
        ../../gcc/cfgexpand.cc:6664

[Bug fortran/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[marxin at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Martin Liška <marxin at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Ever confirmed|0                           |1
                 CC|                            |marxin at gcc dot gnu.org,
                   |                            |tkoenig at gcc dot gnu.org
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2022-10-20

--- Comment #1 from Martin Liška <marxin at gcc dot gnu.org> ---
Likely started with r10-485-gbf09e559b22b44e7.

[Bug fortran/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |10.5
           Priority|P3                          |P2

[Bug fortran/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

anlauf at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |anlauf at gcc dot gnu.org

--- Comment #2 from anlauf at gcc dot gnu.org ---
A few comments: the testcase should rather be a subroutine; it will no longer
fail once we manage to make variables in the main implicit-save... ;-)

So a better testcase:

subroutine p
  real :: a(1:1000000, 200)
  call s (a(1:2000000, :  ))
end

We do not need optimization; the important flag is "-finline-arg-packing".
The other flags (-m32 -fsanitize=address -frecursive) are needed.

Adding a "save :: a" also avoids the ICE.

[Bug fortran/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

--- Comment #3 from Thomas Koenig <tkoenig at gcc dot gnu.org> ---
As this is invalid code (and in Fortran), should this actually be P2?

[Bug fortran/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[tkoenig at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Thomas Koenig <tkoenig at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P2                          |P3

[Bug fortran/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jakub at gcc dot gnu.org
           Priority|P3                          |P4
           Keywords|                            |error-recovery

[Bug fortran/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

--- Comment #4 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Corresponding C testcase:
-m32 -O2 -fsanitize=address

void bar (float *, float *);

void
foo (void)
{
  float a[400000000];
  float b[200000000];
  bar (a, b);
}

[Bug middle-end/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |jakub at gcc dot gnu.org

--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Created attachment 53954
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=53954&action=edit
gcc13-pr107317.patch

Untested fix.

[Bug middle-end/107317] [10/11/12/13 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

--- Comment #6 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jakub Jelinek <jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:b6330a7685476fc30b8ae9bbf3fca1a9b0d4be95

commit r13-4289-gb6330a7685476fc30b8ae9bbf3fca1a9b0d4be95
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Thu Nov 24 11:29:54 2022 +0100

    asan: Fix up error recovery for too large frames [PR107317]

    asan_emit_stack_protection and functions it calls have various asserts that
    verify sanity of the stack protection instrumentation.  But, that
    verification can easily fail if we've diagnosed a frame offset overflow.
    asan_emit_stack_protection just emits some extra code in the prologue,
    if we've reported errors, we aren't producing assembly, so it doesn't
    really matter if we don't include the protection code, compilation
    is going to fail anyway.

    2022-11-24  Jakub Jelinek  <jakub@redhat.com>

            PR middle-end/107317
            * asan.cc: Include diagnostic-core.h.
            (asan_emit_stack_protection): Return NULL early if seen_error ().

            * gcc.dg/asan/pr107317.c: New test.

[Bug middle-end/107317] [10/11/12 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[10/11/12/13 Regression]    |[10/11/12 Regression] ICE
                   |ICE in emit_redzone_byte,   |in emit_redzone_byte, at
                   |at asan.cc:1508             |asan.cc:1508

--- Comment #7 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Fixed on the trunk so far.

[Bug middle-end/107317] [10/11/12 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

--- Comment #8 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-12 branch has been updated by Jakub Jelinek
<jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:ff185dd96ac4576e722b39fc0f7026281de06eb2

commit r12-9122-gff185dd96ac4576e722b39fc0f7026281de06eb2
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Thu Nov 24 11:29:54 2022 +0100

    asan: Fix up error recovery for too large frames [PR107317]

    asan_emit_stack_protection and functions it calls have various asserts that
    verify sanity of the stack protection instrumentation.  But, that
    verification can easily fail if we've diagnosed a frame offset overflow.
    asan_emit_stack_protection just emits some extra code in the prologue,
    if we've reported errors, we aren't producing assembly, so it doesn't
    really matter if we don't include the protection code, compilation
    is going to fail anyway.

    2022-11-24  Jakub Jelinek  <jakub@redhat.com>

            PR middle-end/107317
            * asan.cc: Include diagnostic-core.h.
            (asan_emit_stack_protection): Return NULL early if seen_error ().

            * gcc.dg/asan/pr107317.c: New test.

    (cherry picked from commit b6330a7685476fc30b8ae9bbf3fca1a9b0d4be95)

[Bug middle-end/107317] [10/11 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[10/11/12 Regression] ICE   |[10/11 Regression] ICE in
                   |in emit_redzone_byte, at    |emit_redzone_byte, at
                   |asan.cc:1508                |asan.cc:1508

--- Comment #9 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Fixed for gcc 12.3 too.

[Bug middle-end/107317] [10/11 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

--- Comment #10 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-11 branch has been updated by Jakub Jelinek
<jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:07b397d7720c062d91669d49fbfb644fb3eb81d8

commit r11-10684-g07b397d7720c062d91669d49fbfb644fb3eb81d8
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Thu Nov 24 11:29:54 2022 +0100

    asan: Fix up error recovery for too large frames [PR107317]

    asan_emit_stack_protection and functions it calls have various asserts that
    verify sanity of the stack protection instrumentation.  But, that
    verification can easily fail if we've diagnosed a frame offset overflow.
    asan_emit_stack_protection just emits some extra code in the prologue,
    if we've reported errors, we aren't producing assembly, so it doesn't
    really matter if we don't include the protection code, compilation
    is going to fail anyway.

    2022-11-24  Jakub Jelinek  <jakub@redhat.com>

            PR middle-end/107317
            * asan.c: Include diagnostic-core.h.
            (asan_emit_stack_protection): Return NULL early if seen_error ().

            * gcc.dg/asan/pr107317.c: New test.

    (cherry picked from commit b6330a7685476fc30b8ae9bbf3fca1a9b0d4be95)

[Bug middle-end/107317] [10 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[10/11 Regression] ICE in   |[10 Regression] ICE in
                   |emit_redzone_byte, at       |emit_redzone_byte, at
                   |asan.cc:1508                |asan.cc:1508

--- Comment #11 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Fixed for 11.4 as well.

[Bug middle-end/107317] [10 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

--- Comment #12 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-10 branch has been updated by Jakub Jelinek
<jakub@gcc.gnu.org>:

https://gcc.gnu.org/g:2525ad2b2aa0c2ddc64b091ba8a35c04c47e8723

commit r10-11344-g2525ad2b2aa0c2ddc64b091ba8a35c04c47e8723
Author: Jakub Jelinek <jakub@redhat.com>
Date:   Thu Nov 24 11:29:54 2022 +0100

    asan: Fix up error recovery for too large frames [PR107317]

    asan_emit_stack_protection and functions it calls have various asserts that
    verify sanity of the stack protection instrumentation.  But, that
    verification can easily fail if we've diagnosed a frame offset overflow.
    asan_emit_stack_protection just emits some extra code in the prologue,
    if we've reported errors, we aren't producing assembly, so it doesn't
    really matter if we don't include the protection code, compilation
    is going to fail anyway.

    2022-11-24  Jakub Jelinek  <jakub@redhat.com>

            PR middle-end/107317
            * asan.c: Include diagnostic-core.h.
            (asan_emit_stack_protection): Return NULL early if seen_error ().

            * gcc.dg/asan/pr107317.c: New test.

    (cherry picked from commit b6330a7685476fc30b8ae9bbf3fca1a9b0d4be95)

[Bug middle-end/107317] [10 Regression] ICE in emit_redzone_byte, at asan.cc:1508

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107317

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #13 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Fixed for 10.5 too.