Re: protected alloca class for malloc fallback

[Richard Biener <richard.guenther@gmail.com>]

On Tue, Aug 16, 2016 at 7:54 PM, Martin Sebor <msebor@gmail.com> wrote:
> On 08/16/2016 10:47 AM, Jeff Law wrote:
>>
>> On 08/16/2016 10:44 AM, Jakub Jelinek wrote:
>>>
>>> On Tue, Aug 16, 2016 at 10:27:58AM -0600, Jeff Law wrote:
>>>>
>>>> I think you're being rather short-sighed here.  GCC is being used in
>>>> ways we
>>>> can't necessarily predict -- which might include compile servers,
>>>> JITs, web
>>>> services, etc.
>>>
>>>
>>> For compile server/web services one needs to add the protection
>>> outside of
>>> gcc (sandboxing, containers, SELinux, limiting CPU and/or memory, etc.),
>>> because even with very short testcases e.g. in C/C++ one can eat
>>> arbitrary
>>> amounts of stack even without any uses of alloca in the compiler, simply
>>> through deep recursion in the parsers etc.
>>
>> Agreed.  However, that doesn't mean we should not be locking down things
>> like alloca and other attack vectors.
>
>
> I think I made this suggestion when Aldy posted his first patch
> but it didn't get much traction so let me try again.  Since the
> concern is alloca calls with excessively large arguments, would
> transforming those (determined both at compile time and at run
> time) into pairs of malloc/free calls be an acceptable compromise?
>
> It would seem like a natural complement to the transformation
> in the opposite direction, brought up back then, of turning calls
> to malloc with small (compile-time) arguments into alloca.
>
> I would expect the malloc optimization to more than outweigh
> the performance cost of the alloca to malloc transformation.
> Perhaps even to the point to obviate the need for any explicit
> alloca calls at all.  With the optimization in place, it seems
> that it should even be possible to transparently transform at
> least the most basic uses of some C++ containers written in
> terms of operator new and delete to use alloca instead when
> their sizes were known and under the alloca to malloc threshold.

Please instead work on sth like -fstack-protector for alloca -- it should
be straight-forward to add a runtime test on the stack adjustment
being performed against some magic bound (yeah, needs more than
only GCC support here).

Richard.

> Martin