[Bug tree-optimization/104970] [12 Regression] ICE in execute_todo, at passes.cc:2133 since r12-6480-gea19c8f33a3a8d2b

["msebor at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104970

--- Comment #5 from Martin Sebor <msebor at gcc dot gnu.org> ---
Incidentally, __builtin_dynamic_object_size returns the size of a VLA parameter
in both mininum and maximum modes.  In f0 below, the size of the A array is at
least N bytes but it could be more, so based on my reading of the manual, it
seems like f0 is actually supposed return -1 (f1 looks fine).  If my reading is
correct that would seem unfortunate because it would basically makes BDOS
ineffective for _FORTIFY_SOURCE of VLA parameters.

long f0 (int n, char a[static n])
{
  return __builtin_dynamic_object_size (a, 0);   // folded to n, should be -1?
(Clang folds to -1)
}

long f1 (int n, char a[static n])
{
  return __builtin_dynamic_object_size (a, 1);   // folded to n (Clang folds to
-1)
}

Even more unfortunate seems that that without the [static] it's not undefined
to pass an array with fewer elements than the VLA bound indicates to a function
like f0 of f1.  GCC BDOS doesn't seem to consider the [static] notation and
folds the result the same way either way.  So while well-written code will
benefit from the stricter runtime checking made possible by the tighter bound,
it will cause aborts for poorly written code that's strictly valid.  If I'm
right about this, adding a permissive mode to BDOS to accommodate the poorly
written but valid code might be a way out.

There are cases when __builtin_dynamic_object_size could put the VLA bounds to
use, although I suspect they don't mater for _FORTIFY_SOURCE; if they should
matter, the brute force pr97172 fix might need to be revisited and the bounds
somehow preserved  Here are some such use cases:

$ cat c.c && gcc -O -S c.c
long f0 (int n, char a[static n])
{
  return __builtin_dynamic_object_size (a, 1);   // folded to n
}

long f1 (int n, char (*a)[n])
{
  return __builtin_dynamic_object_size (*a, 1);   // folded to -1 (fold to n?)
}

long f2 (int n, char a[1][n])
{
  return __builtin_dynamic_object_size (a[0], 1);   // folded to -1 (fold to
n?)
}

long f3 (int n, char a[static 1][n])
{
  return __builtin_dynamic_object_size (a, 1);   // ICE (fold to n?)
}