Re: protected alloca class for malloc fallback

[Martin Sebor <msebor@gmail.com>]

On 08/17/2016 02:27 AM, Richard Biener wrote:
> On Tue, Aug 16, 2016 at 7:54 PM, Martin Sebor <msebor@gmail.com> wrote:
>> On 08/16/2016 10:47 AM, Jeff Law wrote:
>>>
>>> On 08/16/2016 10:44 AM, Jakub Jelinek wrote:
>>>>
>>>> On Tue, Aug 16, 2016 at 10:27:58AM -0600, Jeff Law wrote:
>>>>>
>>>>> I think you're being rather short-sighed here.  GCC is being used in
>>>>> ways we
>>>>> can't necessarily predict -- which might include compile servers,
>>>>> JITs, web
>>>>> services, etc.
>>>>
>>>>
>>>> For compile server/web services one needs to add the protection
>>>> outside of
>>>> gcc (sandboxing, containers, SELinux, limiting CPU and/or memory, etc.),
>>>> because even with very short testcases e.g. in C/C++ one can eat
>>>> arbitrary
>>>> amounts of stack even without any uses of alloca in the compiler, simply
>>>> through deep recursion in the parsers etc.
>>>
>>> Agreed.  However, that doesn't mean we should not be locking down things
>>> like alloca and other attack vectors.
>>
>>
>> I think I made this suggestion when Aldy posted his first patch
>> but it didn't get much traction so let me try again.  Since the
>> concern is alloca calls with excessively large arguments, would
>> transforming those (determined both at compile time and at run
>> time) into pairs of malloc/free calls be an acceptable compromise?
>>
>> It would seem like a natural complement to the transformation
>> in the opposite direction, brought up back then, of turning calls
>> to malloc with small (compile-time) arguments into alloca.
>>
>> I would expect the malloc optimization to more than outweigh
>> the performance cost of the alloca to malloc transformation.
>> Perhaps even to the point to obviate the need for any explicit
>> alloca calls at all.  With the optimization in place, it seems
>> that it should even be possible to transparently transform at
>> least the most basic uses of some C++ containers written in
>> terms of operator new and delete to use alloca instead when
>> their sizes were known and under the alloca to malloc threshold.
>
> Please instead work on sth like -fstack-protector for alloca -- it should
> be straight-forward to add a runtime test on the stack adjustment
> being performed against some magic bound (yeah, needs more than
> only GCC support here).

I agree that would be a useful first step, and certainly a more 
attainable goal.  The C++ patch that handled the VLA part (bug
69517) had to be reverted just before 6.1 was released but it's
still on my to do list to add to it a knob to adjust the bound
at runtime and resubmit it.  It should be straightforward to
extend a similar approach to alloca and do the same thing in C
(except with trapping rather than throwing an exception).

Martin