[Bug c/8081] ICE with variably sized types returned from nested functions

["rguenther at suse dot de" <gcc-bugzilla@gcc.gnu.org>]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=8081

--- Comment #26 from rguenther at suse dot de <rguenther at suse dot de> 2012-01-13 09:08:30 UTC ---
On Fri, 13 Jan 2012, ebotcazou at gcc dot gnu.org wrote:

> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=8081
> 
> Eric Botcazou <ebotcazou at gcc dot gnu.org> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |ebotcazou at gcc dot
>                    |                            |gnu.org
> 
> --- Comment #25 from Eric Botcazou <ebotcazou at gcc dot gnu.org> 2012-01-13 08:36:56 UTC ---
> > It's of course not that easy as we gimplify before un-nesting.  The frontend
> > would be responsible to arrange things that way, similar to how we pass
> > a return slot in the C++ frontend (DECL_BY_REFERENCE on the DECL_RESULT
> > variable).  Like for the attached patch.  Passes
> > 
> > extern void abort (void);
> > int
> > main (int argc, char **argv)
> > {
> >   int size = 10;
> >   typedef struct
> >     {
> >       char val[size];
> >     }
> >   block;
> >   block a, b;
> >   block __attribute__((noinline))
> >   retframe_block ()
> >     {
> >       return *(block *) &b;
> >     }
> >   b.val[0] = -1;
> >   b.val[1] = -2;
> >   a=retframe_block ();
> >   if (a.val[0] != -1
> >       || a.val[1] != -2)
> >     abort ();
> >   return 0;
> > }
> > 
> > I'm not sure if one can construct a testcase where using return-slot
> > optimization causes wrong-code generation.  Alternatively checking
> > DECL_BY_REFERENCE on the callees DECL_RESULT instead of applying it to
> > all VLA types could work (though not for indirect calls).
> 
> You should ask specialists. :-)  In Ada, we do this routinely and the strategy
> used is that of the "forced RSO": we generate INIT_EXPR instead of MODIFY_EXPR
> and we create an explicit temporary if we detect potential overlap.
> 
> Btw, I don't understand why you're mixing DECL_BY_REFERENCE and RSO here, just
> 
> Index: gimplify.c
> ===================================================================
> --- gimplify.c  (revision 183104)
> +++ gimplify.c  (working copy)
> @@ -4417,6 +4417,9 @@ gimplify_modify_expr_rhs (tree *expr_p,
>                 /* It's OK to use the target directly if it's being
>                    initialized. */
>                 use_target = true;
> +             else if (variably_modified_type_p (TREE_TYPE (*to_p), NULL_TREE))
> +               /* Always use the target for variable-sized types.  */
> +               use_target = true;
>               else if (TREE_CODE (*to_p) != SSA_NAME
>                       && (!is_gimple_variable (*to_p)
>                           || needs_to_live_in_memory (*to_p)))
> 
> works for me on the testcase.

Ah, ok.  So I suppose the Frontend could force RSO here as well by
just setting CALL_EXPR_RETURN_SLOT_OPT on the CALL_EXPR.  Not sure
which approach is better.