[Bug libgcc/106949] New: Memory leak using VLA with -fsplit-stack

[Bug libgcc/106949] New: Memory leak using VLA with -fsplit-stack

[iam at datacompboy dot ru]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106949

            Bug ID: 106949
           Summary: Memory leak using VLA with -fsplit-stack
           Product: gcc
           Version: 11.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libgcc
          Assignee: unassigned at gcc dot gnu.org
          Reporter: iam at datacompboy dot ru
  Target Milestone: ---

Created attachment 53575
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=53575&action=edit
minimal repro case

Having a function with couple of VLA arrays enough to cause a memory leak.

If you run the function in a loop, memory leak only grows.

```
$ gcc -fsplit-stack -g b.c && valgrind ./a.out
==6861== HEAP SUMMARY:
==6861==     in use at exit: 20,040 bytes in 2 blocks
==6861==   total heap usage: 2 allocs, 0 frees, 20,040 bytes allocated
==6861==
==6861== LEAK SUMMARY:
==6861==    definitely lost: 24 bytes in 1 blocks
==6861==    indirectly lost: 20,016 bytes in 1 blocks
```

With a loop running function 10k times:
```
$ gcc -fsplit-stack -g c.c && valgrind ./a.out
==6931== HEAP SUMMARY:
==6931==     in use at exit: 569,801,800 bytes in 32,162 blocks
==6931==   total heap usage: 32,162 allocs, 0 frees, 569,801,800 bytes
allocated
==6931==
==6931== LEAK SUMMARY:
==6931==    definitely lost: 385,944 bytes in 16,081 blocks
==6931==    indirectly lost: 569,121,904 bytes in 16,072 blocks
==6931==      possibly lost: 293,952 bytes in 9 blocks
```

[Bug libgcc/106949] Memory leak using VLA with -fsplit-stack

[iam at datacompboy dot ru]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106949

--- Comment #1 from Anton Fedorov <iam at datacompboy dot ru> ---
Dunno, is this related to this point?
https://github.com/gcc-mirror/gcc/blob/master/libgcc/generic-morestack.c#L796

  /* If we are still on the initial stack, then we have a space leak.
     FIXME.  */

[Bug libgcc/106949] Memory leak using VLA with -fsplit-stack

[iam at datacompboy dot ru]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106949

--- Comment #2 from Anton Fedorov <iam at datacompboy dot ru> ---
Created attachment 53654
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=53654&action=edit
potential fix

I checked with the HEAD (43faf3e5445b571731e52faa1be085ecd0a09323) and the
issue is still there.

While it's kind of trivial to move the leak from "leak" to "not freed by the
end of the program" with proposed patch to create an empty segment, it doesn't
seems to fix the problem with the approach that there is no way to mark
malloc()'ed alloca()'s in case we are on the main stack segment.

But the problem extends beyond the allocation on the main segment -- no matter
when we enter the some large function that keeps calling the subfunction with
VLA/alloca, the memory will keep growing without collection.

For example, if we'll call leak function in a loop (imagine we run even
processing loop, network processing loop etc etc that calls some handler that
have alloca/VLA), the memory will grow without any way to free it.

```
static int leak(int s) {
  volatile int n[5000+s];

  for (int i = 0; i < s; i++) {
    n[i] = 0;
  }
  return -1;
}
int main() {
  int a = leak(1); a += a;
  for(int i = 0; i < 10000; ++i)
    a += leak(i+1);
  return 0;
}
```

I see the option to generate code to explicitly move N (number of calls to
__morestack_allocate_stack_space in the function) first dynamic_allocation
elements to free_dynamic_allocation list before execution return -- at cost of
extra check & jmp for case when stack is sufficient.

Given that -fsplit-stack accepts extra cost for stack guarantees -- this extra
cost for avoiding memory leak should be fine?

[Bug libgcc/106949] Memory leak using VLA with -fsplit-stack

[ian at airs dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106949

--- Comment #3 from Ian Lance Taylor <ian at airs dot com> ---
I don't think your attached patch is going to work.  The code assumes that it
is running within a stack segment.  You can't just add a stack segment without
changing the stack pointer.

But something like your suggestion might work.  If the function is going to
call __morestack_allocate_stack_space, then at the start of the function call a
new function __morestack_allocating_stack_space.  That function can return a
pointer.  The __morestack_allocate_stack_space function can add its allocations
to a list at that pointer.  At the end of the function call another new
function that releases the allocated space.  Some work will be required to make
sure that the space gets released if an exception is thrown.

[Bug libgcc/106949] Memory leak using VLA/alloca with -fsplit-stack

[iam at datacompboy dot ru]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106949

--- Comment #4 from Anton Fedorov <iam at datacompboy dot ru> ---
> You can't just add a stack segment without changing the stack pointer.

I can -- since we are on the initial stack at this point, no return into
__morestack will happen so no attempt to release it, thus there is no problem
and it won't affect subsequent allocations of frames would it be necessary.

But that's the catch: the allocated blocks won't be released no matter is we on
the initial stack or deep inside, until we try to release the frame, which may
never happen no matter are we on the initial stack or not.

the easiest way may be change gcc generate the code like:

```
struct __morestack_blocks {
  struct __morestack_blocks * next;
  void * block;
};

void __morestack_free_blocks(struct __morestack_blocks ** blocks) {
  struct __morestack_blocks * cur = *blocks;
  while(cur) {
    free(cur->block);
    cur = cur->next;
  }
}

void* __morestack_allocate_stack_space(int size, struct __morestack_blocks **
blocks){ ... }

somefunction(someargs) {
  struct __morestack_blocks __attribute__((cleanup(__morestack_free_blocks))) =
nullptr;

  // void * a = alloca(x);
  a = __morestack_allocate_stack_space(x, &__morestack_blocks);

}
```

But that local variable to be used as a reference, but makes it trivial
(although, I don't know how to generate local variable with assigned destructor
from the gcc code, will look for it)...

[Bug libgcc/106949] Memory leak using VLA/alloca with -fsplit-stack

[iam at datacompboy dot ru]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106949

--- Comment #5 from Anton Fedorov <iam at datacompboy dot ru> ---
Hhhhmmm... Interesting, it doesn't seems to be possible, but it would be fun
way to fix to add cleanup(free) attribute to the variable if it has to be
malloc'ed...