[Bug c/68065] Size calculations for VLAs can overflow

["joseph at codesourcery dot com" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68065

--- Comment #11 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
On Wed, 28 Oct 2015, ch3root at openwall dot com wrote:

> --- Comment #10 from Alexander Cherepanov <ch3root at openwall dot com> ---
> On 2015-10-27 20:09, joseph at codesourcery dot com wrote:
> > I think it's undefined at the point where a type exceeds the limit on the
> > size of an object
> 
> This would probably be the most reasonable approach but it's not clear 
> if the text of the standard supports it. E.g., the list of UB (C11, 
> J.2p1) includes this one:
> 
>    - The size expression in an array declaration is not a constant 
> expression and evaluates at program execution time to a nonpositive 
> value (6.7.6.2).
> 
> but I don't see anything like what you described. Perhaps I'm missing 
> something?

The standard has a list of minimum implementation limits, but it is not 
exhaustive as to all areas in which there may be limits.  I think it is 
reasonable to consider "bytes in an object" to cover such limits for any 
object type, whether or not an object of that type is constructed.

It is well-known that strictly conforming programs do not exist when you 
interpret the standard strictly.

The existence of rsize_t in Annex K hints at the likelihood of problems 
with objects larger than SIZE_MAX >> 1 bytes.

> This is also very nice approach. (Although it seems to differ from the 
> approach to non-VLA arrays in 
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68107#c1 .) But, again, I 

It's universally the case that size-related things that are constraint 
violations (compile-time errors) for non-VLAs are runtime undefined 
behavior for VLAs.  This applies to size overflow just as to e.g. 
non-positive array sizes.

There are also lots of details around evaluation of VLA sizes that the C 
standard is silent on (the C standard doesn't really have a notion of type 
names being evaluated at all, only expressions, but VLAs in type names 
only make sense when type names get evaluated at an appropriate point in 
execution) that I had to fill out when sorting out constant expressions 
and VLAs issues for GCC 4.5.