[Bug tree-optimization/96051] New: alloca(0) triggers TCO for VLA

[Bug tree-optimization/96051] New: alloca(0) triggers TCO for VLA

[fwd at quantentunnel dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96051

            Bug ID: 96051
           Summary: alloca(0) triggers TCO for VLA
           Product: gcc
           Version: 8.3.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: fwd at quantentunnel dot de
  Target Milestone: ---

Created attachment 48831
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=48831&action=edit
TCO with alloca(0)

I have written a function which is properly translated into tail-call-optimized
code.
Then I tried to replace a call to alloca inside that function by a VLA.
Doing so I found a strange behaviour.

Using the VLA in place of alloca, but leaving the statement '(int*)alloca(0)'
in the source code, still triggers TCO. Removing the (unused) call to alloca(0)
prevents the TCO.
It is weird because I expected that alloca(0) is removed because it should have
no effect.

Summary:
1. alloca(n); // n > 0; works as expected (TCO)
2. using VLA and alloca(0); the same output (assembler) as in case 1 (TCO)
3. using VLA without alloca; works as expected (no TCO)


Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 8.3.0-6'
--with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs
--enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr
--with-gcc-major-version-only --program-suffix=-8
--program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
--libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu
--enable-libstdcxx-debug --enable-libstdcxx-time=yes
--with-default-libstdcxx-abi=new --enable-gnu-unique-object
--disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie
--with-system-zlib --with-target-system-zlib --enable-objc-gc=auto
--enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64
--with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic
--enable-offload-targets=nvptx-none --without-cuda-driver
--enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu
--target=x86_64-linux-gnu
Thread model: posix
gcc version 8.3.0 (Debian 8.3.0-6)

[Bug tree-optimization/96051] alloca(0) triggers TCO for VLA

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96051

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement
   Last reconfirmed|                            |2021-12-05
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |NEW

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
  /* If the code both contains VLAs and calls alloca, then we cannot reclaim
     the stack space allocated to the VLAs.  */

And the stack restore in the case of VLA without the alloca just happens to be
after the recusive function call to no_tco_with_vla_only.

To do this optimization, we need to move the stack restore before the recusive
function call and for that we need to make sure the VLA address never escapes.

Anyways Confirmed.