[Bug c++/107360] New: ICE on sizeof(*f(x)) when f's (deduced) return type is a pointer to VLA

[Bug c++/107360] New: ICE on sizeof(*f(x)) when f's (deduced) return type is a pointer to VLA

[izbyshev at ispras dot ru]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107360

            Bug ID: 107360
           Summary: ICE on sizeof(*f(x)) when f's (deduced) return type is
                    a pointer to VLA
           Product: gcc
           Version: 12.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: izbyshev at ispras dot ru
  Target Milestone: ---

auto f(int n) {
    int (*a)[n] = 0;
    return a;
}

int g() {
    return sizeof *f(1);
}

Output of GCC 12.2:

during RTL pass: expand
<source>: In function 'int g()':
<source>:7:23: internal compiler error: in expand_expr_real_1, at expr.cc:10586
    7 |     return sizeof *f(1);
      |                       ^
0x1bb069e internal_error(char const*, ...)
        ???:0
0x6ff396 fancy_abort(char const*, int, char const*)
        ???:0
0xab97bd expand_expr_real_2(separate_ops*, rtx_def*, machine_mode,
expand_modifier)
        ???:0
0xaab144 expand_expr_real_1(tree_node*, rtx_def*, machine_mode,
expand_modifier, rtx_def**, bool)
        ???:0
0xab97bd expand_expr_real_2(separate_ops*, rtx_def*, machine_mode,
expand_modifier)
        ???:0
0xaab144 expand_expr_real_1(tree_node*, rtx_def*, machine_mode,
expand_modifier, rtx_def**, bool)
        ???:0
0xab38c4 expand_operands(tree_node*, tree_node*, rtx_def*, rtx_def**,
rtx_def**, expand_modifier)
        ???:0
0xaba7ac expand_expr_real_2(separate_ops*, rtx_def*, machine_mode,
expand_modifier)
        ???:0
0xaab144 expand_expr_real_1(tree_node*, rtx_def*, machine_mode,
expand_modifier, rtx_def**, bool)
        ???:0
0xabb2c9 expand_expr_real_2(separate_ops*, rtx_def*, machine_mode,
expand_modifier)
        ???:0
0xaab144 expand_expr_real_1(tree_node*, rtx_def*, machine_mode,
expand_modifier, rtx_def**, bool)
        ???:0

All versions supporting C++14 return type deduction (i.e. since 4.8) produce
ICE, except 4.9, which generates wrong code for g() instead (it returns 1
regardless of f's argument).

[Bug c++/107360] ICE on sizeof(*f(x)) when f's (deduced) return type is a pointer to VLA

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107360

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|ice-on-valid-code           |

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Maybe this should be invalid code ...

[Bug c++/107360] ICE on sizeof(*f(x)) when f's (deduced) return type is a pointer to VLA

[izbyshev at ispras dot ru]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107360

--- Comment #2 from Alexey Izbyshev <izbyshev at ispras dot ru> ---
(In reply to Andrew Pinski from comment #1)
> Maybe this should be invalid code ...

Yes, I think it should be invalid. VLAs are not allowed in function return
types in C. VLAs in C++ are a GCC extension, but it's unclear to me why (and
how) GCC should allow them in return types, even when they don't depend on
function parameters.

Currently, GCC crashes even in the latter case, e.g.:

extern int n;

auto f() {
    int (*a)[n] = 0;
    return a;
}

int g() {
    return sizeof *f();
}

[Bug c++/107360] ICE on sizeof(*f(x)) when f's (deduced) return type is a pointer to VLA

[muecker at gwdg dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107360

Martin Uecker <muecker at gwdg dot de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |muecker at gwdg dot de

--- Comment #3 from Martin Uecker <muecker at gwdg dot de> ---

This is likely related to PR108423 and PR107557.

I think there there are cases were variably modified
return types are allowed in ISO C:

void f(int n, double (*(bar(void)))[n])
{
    double (*p)[n] = bar();
}

[Bug c++/107360] ICE on sizeof(*f(x)) when f's (deduced) return type is a pointer to VLA

[izbyshev at ispras dot ru]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107360

--- Comment #4 from Alexey Izbyshev <izbyshev at ispras dot ru> ---
(In reply to Martin Uecker from comment #3)
> I think there there are cases were variably modified
> return types are allowed in ISO C:
> 
> void f(int n, double (*(bar(void)))[n])
> {
>     double (*p)[n] = bar();
> }

Yes, I should have been more precise. Per C11 6.7.6.2:

> If an identifier is declared as having a variably modified type, it shall be an ordinary identifier (as defined in 6.2.3), have no linkage, and have either block scope or function prototype scope.

So, since my original example is in C++ and involves return type deduction,
what I really wanted to say is that it's not allowed to *define* a function
with a variably modified return type in ISO C (hence it seems fine to not
support it in GCC's VLA extension for C++ too and consider such code invalid).

GCC does allow to define a nested function with a variably modified return type
though, and, perhaps surprisingly, a test case similar to my original C++ one
is compiled successfully:

extern int n;

int f() {
  int a[n];
  int (*g())[n] {
    return &a;
  };
  return sizeof *g();
}

But what would seem to be a C++ equivalent crashes GCC:

extern int n;

int f() {
  int a[n];
  auto g = [&]() {
    return &a;
  };
  return sizeof *g();
}

during RTL pass: expand
<source>: In function 'int f()':
<source>:6:13: internal compiler error: in expand_expr_real_1, at expr.cc:10586
    6 |     return &a;
      |             ^
...

And for this one it's less clear to me whether GCC wants to consider it
invalid.

But without return type deduction GCC refuses to compile:

extern int n;

int f() {
  typedef int t[n];
  t a;
  auto g = [&]() -> t* {
    return &a;
  };
  return sizeof *g();
}

<source>: In function 'int f()':
<source>:6:12: error: data member may not have variably modified type 'int
(*())[n]'
    6 |   auto g = [&]() -> t* {
      |            ^
<source>:6:12: error: declaration of 'operator()' as non-function
Compiler returned: 1

[Bug c++/107360] ICE on sizeof(*f(x)) when f's (deduced) return type is a pointer to VLA

[muecker at gwdg dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107360

--- Comment #5 from Martin Uecker <muecker at gwdg dot de> ---

Yes. After reconsidering this, this is not related to the other bugs mentioned
above. I think your first example needs to be invalid, because the derived
return type then depends on a variable inside the nested functions, which can
not work without new pointer types.

The second example could work.