[Bug c/8081] ICE with variably sized types returned from nested functions

["rguenth at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=8081

--- Comment #24 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-01-12 10:23:05 UTC ---
Created attachment 26306
  --> http://gcc.gnu.org/bugzilla/attachment.cgi?id=26306
A patch for using by-reference passing

(In reply to comment #23)
> as alternative to rejecting this case we can lower returning variable-size
> types during un-nesting to explicit passing of a return slot and using
> memcpy, making the nested function return nothing.

It's of course not that easy as we gimplify before un-nesting.  The frontend
would be responsible to arrange things that way, similar to how we pass
a return slot in the C++ frontend (DECL_BY_REFERENCE on the DECL_RESULT
variable).  Like for the attached patch.  Passes

extern void abort (void);
int
main (int argc, char **argv)
{
  int size = 10;
  typedef struct
    {
      char val[size];
    }
  block;
  block a, b;
  block __attribute__((noinline))
  retframe_block ()
    {
      return *(block *) &b;
    }
  b.val[0] = -1;
  b.val[1] = -2;
  a=retframe_block ();
  if (a.val[0] != -1
      || a.val[1] != -2)
    abort ();
  return 0;
}

I'm not sure if one can construct a testcase where using return-slot
optimization causes wrong-code generation.  Alternatively checking
DECL_BY_REFERENCE on the callees DECL_RESULT instead of applying it to
all VLA types could work (though not for indirect calls).