[Bug tree-optimization/105051] New: consider not combining malloc + memset to calloc when inside calloc itself

[Bug tree-optimization/105051] New: consider not combining malloc + memset to calloc when inside calloc itself

[godmar at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105051

            Bug ID: 105051
           Summary: consider not combining malloc + memset to calloc when
                    inside calloc itself
           Product: gcc
           Version: 12.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: godmar at gmail dot com
  Target Milestone: ---

gcc 8.5.0 (and later) just broke our 18-year-old calloc implementation, which
goes like this:

void *
calloc (size_t a, size_t b)
{
  /* Calculate block size and make sure it fits in size_t. */
  size_t size = a * b;

  /* Allocate and zero memory. */
  void *p = malloc (size);
  if (p != NULL)
    memset (p, 0, size);

  return p;
}

by producing this ingenuous code sequence:

calloc:
    imulq   %rsi, %rdi
    movl    $1, %esi
    jmp calloc


I believe that this is due to an optimization whereby a sequence of malloc
followed by memset is transformed into a call to calloc.

My suggestion would be to suppress this optimization inside a function that is
itself called calloc as it leads to an infinite loop.

I'm also curious what the proper work-around is.  Is it `-fno-builtin-memset`?

I discovered this in 8.5.0 but Godbolt says it's present in the trunk as of
now:
https://godbolt.org/z/MKMT98qnr

This may be related to Bug #83022

[Bug tree-optimization/105051] consider not combining malloc + memset to calloc when inside calloc itself

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105051

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |DUPLICATE

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Dup of bug 78074.

Also the code to do this has been there since 2014 (8 years ago) so your 18
year old calloc implementation has been broken for 8 years and it was 10 year
at the time it was broken.

*** This bug has been marked as a duplicate of bug 78074 ***

[Bug tree-optimization/105051] consider not combining malloc + memset to calloc when inside calloc itself

[godmar at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105051

--- Comment #2 from Godmar Back <godmar at gmail dot com> ---
Thank you for your reply.  

To make sure I understand, the only work-around is to completely disable all
builtins (as in -fno-builtin), or is using `-fno-builtin-memset` as I proposed
sufficient?

I'm not sure why this bug is invalid, either. To me, applying this optimization
inside `calloc` seems invalid.

[Bug tree-optimization/105051] consider not combining malloc + memset to calloc when inside calloc itself

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105051

--- Comment #3 from Richard Biener <rguenth at gcc dot gnu.org> ---
It's more of a QOI issue, we have similar (old) bugs where we pattern-match a
memcpy loop to memcpy inside memcpy (PR56888).  The difficulty is to reliably
detect whether we're in a place where emitting a call to 'foo' will recurse. 
Unfortunately the people with knowledge in this area are too busy and given
workarounds exist this hasn't been top priority.

[Bug tree-optimization/105051] consider not combining malloc + memset to calloc when inside calloc itself

[godmar at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105051

--- Comment #4 from Godmar Back <godmar at gmail dot com> ---
Don't let perfect be the enemy of good.  IMO, the detection doesn't need to be
perfect in the sense of not having false negatives.  All 3 bug reports of the
calloc implementations you broke called malloc + memset directly in calloc,
something that (I'd guess?) should be more readily detectable.