[Bug tree-optimization/57742] memset(malloc(n),0,n) -> calloc(n,1)

["rguenth at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=57742

--- Comment #9 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to Marc Glisse from comment #7)
> (In reply to Richard Biener from comment #5)
> > We have walk_aliased_vdefs for this.  Basically the first callback
> > you receive has to be the malloc, otherwise there is an aliasing
> > stmt inbetween.  Initialize the ao_ref with ao_ref_init_from_ptr_and_size.
> 
> Hmm, there is a problem with that: I don't get a callback for malloc.
> stmt_may_clobber_ref_p_1 only looks at the lhs of a call statement if it
> isn't an SSA_NAME, so it considers that p=malloc(n) does not clobber
> MEM_REF[p]. This kind of makes sense, it creates this memory, which is
> different from clobbering. I can look at the def_stmt of the first argument
> of memset to find the malloc, at least, but that doesn't help me with the
> memory checks.
> 
> Also, for this testcase:
> void* f(int n,double*d){
>   int* p=__builtin_malloc(n);
>   ++*d;
>   __builtin_memset(p,0,n);
>   return p;
> }
> I actually get a callback for the store in *d, which gcc believes might
> alias :-(

Yeah well, either because of pass placement or because of points-to
analysis being not context sensitive.

> For this example:
> void g(int*);
> void* f(int n){
>   int* p=__builtin_malloc(n);
>   for(int i=0;i<10000;++i){
>     __builtin_memset(p,0,n);
>     g(p);
>     p[5]=10;
>   }
>   return p;
> }
> if I modify the aliasing machinery to make it believe that p=malloc does
> alias, malloc is the first callback. I haven't added the dominance checks,
> but I assume they will tell me that malloc dominates memset and memset
> postdominates malloc, although I still shouldn't do the transformation.
> 
> Pretty depressed at this point...

Nobody said it was going to be trivial ;)

Exact pattern matching of the CFG involved might be the easiest, plus
manually implementing walk_aliased_vdefs by simply walking the use-def
chain of the virtual operands from the memset operation to the malloc
and checking stmt_may_clobber_ref_p_1 on the ao_ref_init_from_ptr_and_size
ref.