[Bug c/68065] Size calculations for VLAs can overflow

["ch3root at openwall dot com" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68065

--- Comment #7 from Alexander Cherepanov <ch3root at openwall dot com> ---
On 2015-10-27 03:15, joseph at codesourcery dot com wrote:
>> VLA size overflow is very similar to overflow in "new". Shouldn't it be
>> handled in a similar way?
>
> I'm thinking of it as essentially like stack overflow,

The problem is not limited to stack allocations. The example in comment 
#3 works for VLAs as well. This would a quite elegant way to deal with 
untrusted sizes but it seem you cannot get away without any arithmetic 
check at all:-( Maybe saturated arithmetic for sizes could help but 
that's another can of worms.

Well, the problem in non-stack case seems to be in sizeof. My take: 
sizeof is an operator and C11, 6.5p5 applies to it:

   If an exceptional condition occurs during the evaluation of
   an expression (that is, if the result is not mathematically
   defined or not in the range of representable values for its
   type), the behavior is undefined.

That is, an oversized VLA is fine but taking its sizeof is UB. This is 
somewhat similar to how pointer difference is UB when outside ptrdiff_t 
range.

There could be an argument that sizeof works with unsigned types and 
hence should wrap per C11, 6.2.5p9:

   A computation involving unsigned operands can never overflow,
   because a result that cannot be represented by the resulting
   unsigned integer type is reduced modulo the number that is
   one greater than the largest value that can be represented
   by the resulting type.

But sizeof doesn't directly compute anything with its operand (even if 
it's unsigned). Therefore this rule is not applicable.

It seems I can convince myself that non-stack case is Ok:-)

> where it's
> traditionally been the user's job to bound their stack allocations.

Yes, and it's a pity that there is no diagnostics for stack allocation 
problems by default given it's not UB and the code could be conforming 
according to the C standards.

> I think ubsan should enable all of (VLA size overflow checks,

Yes, catching overflows in "sizeof(int[size])" is definitely nice. But 
catching "_Alignof(int[size])" etc. for huge "size" is probably fine too.

> stack checking
> for fixed-size allocations to ensure the amount of stack space allocated
> in one go is small enough that overflow is guaranteed to be detected,
> similar checks for variable size allocations whether from VLAs or alloca).
> Of course separate options for various cases may make sense as well.

It's not practiced in gcc to track such things via the bug tracker?