operator new[] overflow (PR 19351)

operator new[] overflow (PR 19351)

[Florian Weimer]

Mozilla seems to receive a report of an exploitable operator new[]
overflow every couple of months now.  Obviously, this is not good.

What is necessary so that GCC can fix this code generation issue?
I've created a patch, together with a test case, but it has not been
approved, nor have I been told how to change the patch to make it more
suitable for inclusion ("change the middle end type system so that
this can be expressed in a better way" is just not realistic for me,
and apparently anyone else):

  <http://gcc.gnu.org/ml/gcc-patches/2010-02/msg00275.html>

So how can we fix this, more than eight years after it was reported as
a security issue, more than ten years after the defect in the standard
was identified, and more than twenty years after it was introduced
into GCC?

Re: operator new[] overflow (PR 19351)

[Joseph S. Myers]

On Tue, 30 Nov 2010, Florian Weimer wrote:

> Mozilla seems to receive a report of an exploitable operator new[]
> overflow every couple of months now.  Obviously, this is not good.
> 
> What is necessary so that GCC can fix this code generation issue?
> I've created a patch, together with a test case, but it has not been
> approved, nor have I been told how to change the patch to make it more
> suitable for inclusion ("change the middle end type system so that
> this can be expressed in a better way" is just not realistic for me,
> and apparently anyone else):
> 
>   <http://gcc.gnu.org/ml/gcc-patches/2010-02/msg00275.html>

My suggestion at <http://gcc.gnu.org/ml/gcc-patches/2010-02/msg00315.html> 
wasn't to change the type system; it was to add new GENERIC / GIMPLE codes 
within the existing type system.

Saturating arithmetic keeps coming up in one context or another.  It's 
generally relevant if you want to exploit various instructions on various 
processors - or if you want to support vector intrinsics (which may 
include saturating operations) even when the vector instructions aren't 
present (see recent discussions of doing that for SSE intrinsics, for 
example).  Sooner or later I expect someone will decide to do the work for 
at least one target.

Related to the "operator new[]" issue, I'm concerned that saturation alone 
may not be enough to solve problems with overflows in allocation.  If on a 
32-bit system you allocate 2GB or more of memory, then differences between 
addresses in / just after that array cannot be calculated reliably as they 
may excess the range of ptrdiff_t.  Thus actually malloc ought to reject 
allocations that take half or more of the address space, to avoid 
undefined behavior arising in the calling code (such allocations simply 
cannot be used reliably in C or C++).  But I don't see anything in glibc's 
malloc to do so; it seems quite possible that it could allocate over 2GB 
in a single allocation on a 32-bit system.  (See PR 45779, but I think the 
only sensible place to fix this is the implementation of malloc.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: operator new[] overflow (PR 19351)

[Gabriel Dos Reis]

On Tue, Nov 30, 2010 at 3:38 PM, Joseph S. Myers
<joseph@codesourcery.com> wrote:
> On Tue, 30 Nov 2010, Florian Weimer wrote:
>
>> Mozilla seems to receive a report of an exploitable operator new[]
>> overflow every couple of months now.  Obviously, this is not good.
>>
>> What is necessary so that GCC can fix this code generation issue?
>> I've created a patch, together with a test case, but it has not been
>> approved, nor have I been told how to change the patch to make it more
>> suitable for inclusion ("change the middle end type system so that
>> this can be expressed in a better way" is just not realistic for me,
>> and apparently anyone else):
>>
>>   <http://gcc.gnu.org/ml/gcc-patches/2010-02/msg00275.html>
>
> My suggestion at <http://gcc.gnu.org/ml/gcc-patches/2010-02/msg00315.html>
> wasn't to change the type system; it was to add new GENERIC / GIMPLE codes
> within the existing type system.
>
> Saturating arithmetic keeps coming up in one context or another.  It's
> generally relevant if you want to exploit various instructions on various
> processors - or if you want to support vector intrinsics (which may
> include saturating operations) even when the vector instructions aren't
> present (see recent discussions of doing that for SSE intrinsics, for
> example).  Sooner or later I expect someone will decide to do the work for
> at least one target.
>
> Related to the "operator new[]" issue, I'm concerned that saturation alone
> may not be enough to solve problems with overflows in allocation.  If on a
> 32-bit system you allocate 2GB or more of memory, then differences between
> addresses in / just after that array cannot be calculated reliably as they
> may excess the range of ptrdiff_t.  Thus actually malloc ought to reject
> allocations that take half or more of the address space, to avoid
> undefined behavior arising in the calling code (such allocations simply
> cannot be used reliably in C or C++).  But I don't see anything in glibc's
> malloc to do so; it seems quite possible that it could allocate over 2GB
> in a single allocation on a 32-bit system.  (See PR 45779, but I think the
> only sensible place to fix this is the implementation of malloc.)
>


The existing GCC behaviour is a bit more perverse than the
C malloc() case as in

       new T[n]

there is no multiplication that could be credited to careless programmer.
The multiplication is introduced by GCC.

-- Gaby

Re: operator new[] overflow (PR 19351)

[Chris Lattner]

On Nov 30, 2010, at 3:12 PM, Joe Buck wrote:

> On Tue, Nov 30, 2010 at 01:49:23PM -0800, Gabriel Dos Reis wrote:
>> The existing GCC behaviour is a bit more perverse than the
>> C malloc() case as in
>> 
>>       new T[n]
>> 
>> there is no multiplication that could be credited to careless programmer.
>> The multiplication is introduced by GCC.
> 
> ... which suggests strongly that GCC should fix it.  Too bad the ABI is
> frozen; if the internal ABI kept the two values (the size of the type, and
> the number of values) separate and passed two arguments to the allocation
> function, it would be easy to do the right thing (through bad_alloc if the
> multiplication overflows).

You don't need any ABI changes to support this.  For example, clang compiles:

int *foo(long X) {
  return new int[X];
}

into:

__Z3fool:                               ## @_Z3fool
Leh_func_begin0:
## BB#0:                                ## %entry
	movl	$4, %ecx
	movq	%rdi, %rax
	mulq	%rcx
	testq	%rdx, %rdx
	movq	$-1, %rdi
	cmoveq	%rax, %rdi
	jmp	__Znam

On overflow it just forces the size passed in to operator new to -1ULL, which throws bad_alloc.

-Chris

Re: operator new[] overflow (PR 19351)

[Gabriel Dos Reis]

On Wed, Dec 1, 2010 at 5:36 PM, Chris Lattner <clattner@apple.com> wrote:
>
> On Nov 30, 2010, at 3:12 PM, Joe Buck wrote:
>
>> On Tue, Nov 30, 2010 at 01:49:23PM -0800, Gabriel Dos Reis wrote:
>>> The existing GCC behaviour is a bit more perverse than the
>>> C malloc() case as in
>>>
>>>       new T[n]
>>>
>>> there is no multiplication that could be credited to careless programmer.
>>> The multiplication is introduced by GCC.
>>
>> ... which suggests strongly that GCC should fix it.  Too bad the ABI is
>> frozen; if the internal ABI kept the two values (the size of the type, and
>> the number of values) separate and passed two arguments to the allocation
>> function, it would be easy to do the right thing (through bad_alloc if the
>> multiplication overflows).
>
> You don't need any ABI changes to support this.  For example, clang compiles:
>
> int *foo(long X) {
>  return new int[X];
> }
>
> into:
>
> __Z3fool:                               ## @_Z3fool
> Leh_func_begin0:
> ## BB#0:                                ## %entry
>        movl    $4, %ecx
>        movq    %rdi, %rax
>        mulq    %rcx
>        testq   %rdx, %rdx
>        movq    $-1, %rdi
>        cmoveq  %rax, %rdi
>        jmp     __Znam
>
> On overflow it just forces the size passed in to operator new to -1ULL, which throws bad_alloc.

This is a very good point.  At the minimum, GCC should generate similar code
if not improve on it.

-- Gaby

Re: operator new[] overflow (PR 19351)

[Florian Weimer]

* Chris Lattner:

> On overflow it just forces the size passed in to operator new to
> -1ULL, which throws bad_alloc.

This is also what my patch tries to implement.

Re: operator new[] overflow (PR 19351)

[Joe Buck]

On Wed, Dec 01, 2010 at 10:26:58PM -0800, Florian Weimer wrote:
> * Chris Lattner:
> 
> > On overflow it just forces the size passed in to operator new to
> > -1ULL, which throws bad_alloc.
> 
> This is also what my patch tries to implement.

Yes, but Chris's code just checks the overflow of the multiply.  Your
patch achieves the same result in a more complex way, by
computing the largest non-overflowing value of n in

new T[n];

and comparing n against that.  Even though max_size_t/sizeof T is a
compile-time constant, this is still more expensive.

Re: operator new[] overflow (PR 19351)

[Gabriel Dos Reis]

On Thu, Dec 2, 2010 at 2:20 PM, Joe Buck <Joe.Buck@synopsys.com> wrote:
> On Wed, Dec 01, 2010 at 10:26:58PM -0800, Florian Weimer wrote:
>> * Chris Lattner:
>>
>> > On overflow it just forces the size passed in to operator new to
>> > -1ULL, which throws bad_alloc.
>>
>> This is also what my patch tries to implement.
>
> Yes, but Chris's code just checks the overflow of the multiply.  Your
> patch achieves the same result in a more complex way, by
> computing the largest non-overflowing value of n in
>
> new T[n];
>
> and comparing n against that.  Even though max_size_t/sizeof T is a
> compile-time constant, this is still more expensive.

I would expect max_size_t/sizeof(T) to be actually an integer
constant that n is compared against.  I would be surprised
if that one-time comparison is noticeable in real applications that
new an array of objects.

>
>

Re: operator new[] overflow (PR 19351)

[Joe Buck]

On Thu, Dec 02, 2010 at 02:47:30PM -0800, Gabriel Dos Reis wrote:
> On Thu, Dec 2, 2010 at 2:20 PM, Joe Buck <Joe.Buck@synopsys.com> wrote:
> > On Wed, Dec 01, 2010 at 10:26:58PM -0800, Florian Weimer wrote:
> >> * Chris Lattner:
> >>
> >> > On overflow it just forces the size passed in to operator new to
> >> > -1ULL, which throws bad_alloc.
> >>
> >> This is also what my patch tries to implement.
> >
> > Yes, but Chris's code just checks the overflow of the multiply.  Your
> > patch achieves the same result in a more complex way, by
> > computing the largest non-overflowing value of n in
> >
> > new T[n];
> >
> > and comparing n against that.  Even though max_size_t/sizeof T is a
> > compile-time constant, this is still more expensive.
> 
> I would expect max_size_t/sizeof(T) to be actually an integer
> constant that n is compared against.  I would be surprised
> if that one-time comparison is noticeable in real applications that
> new an array of objects.

It's wasted code if the multiply instruction detects the overflow.
It's true that the cost is small (maybe just one extra instruction
and the same number of tests, maybe one more on architectures where you
have to load a large constant), but it is slightly worse code than what
Chris Lattner showed.  Still, it's certainly an improvement on the current
situation and the cost is negligible compared to the call to the
allocator.  Since it's a security issue, some form of the patch should
go in.

Re: operator new[] overflow (PR 19351)

[Florian Weimer]

* Joe Buck:

> It's wasted code if the multiply instruction detects the overflow.
> It's true that the cost is small (maybe just one extra instruction
> and the same number of tests, maybe one more on architectures where you
> have to load a large constant), but it is slightly worse code than what
> Chris Lattner showed.

It's possible to improve slightly on the LLVM code by using the
overflow flag (at least on i386/amd64), as explained in this blog
post:

<http://blogs.msdn.com/b/michael_howard/archive/2005/12/06/500629.aspx>

My patch emits a run-time division if a VLA is used in an allocator.
But that's a semi-deprecated GCC extension, so I don't think we need
to care.

> Still, it's certainly an improvement on the current
> situation and the cost is negligible compared to the call to the
> allocator.  Since it's a security issue, some form of the patch should
> go in.

Well, should I resubmit, with the fix for the problem building
size_t(-1)?

Re: operator new[] overflow (PR 19351)

[Gabriel Dos Reis]

On Sat, Dec 4, 2010 at 7:22 AM, Florian Weimer <fw@deneb.enyo.de> wrote:
> * Joe Buck:
>
>> It's wasted code if the multiply instruction detects the overflow.
>> It's true that the cost is small (maybe just one extra instruction
>> and the same number of tests, maybe one more on architectures where you
>> have to load a large constant), but it is slightly worse code than what
>> Chris Lattner showed.
>
> It's possible to improve slightly on the LLVM code by using the
> overflow flag (at least on i386/amd64), as explained in this blog
> post:
>
> <http://blogs.msdn.com/b/michael_howard/archive/2005/12/06/500629.aspx>
>
> My patch emits a run-time division if a VLA is used in an allocator.
> But that's a semi-deprecated GCC extension, so I don't think we need
> to care.

Personally, the VLA issue is not one I would care much about.
If it can be done without much cost, fine.  Otherwise, I would
not tie the checking of the standard construct to it.

>
>> Still, it's certainly an improvement on the current
>> situation and the cost is negligible compared to the call to the
>> allocator.  Since it's a security issue, some form of the patch should
>> go in.
>
> Well, should I resubmit, with the fix for the problem building
> size_t(-1)?

I think that would help.

-- Gaby

Re: operator new[] overflow (PR 19351)

[Chris Lattner]

On Dec 4, 2010, at 5:22 AM, Florian Weimer wrote:

> * Joe Buck:
> 
>> It's wasted code if the multiply instruction detects the overflow.
>> It's true that the cost is small (maybe just one extra instruction
>> and the same number of tests, maybe one more on architectures where you
>> have to load a large constant), but it is slightly worse code than what
>> Chris Lattner showed.
> 
> It's possible to improve slightly on the LLVM code by using the
> overflow flag (at least on i386/amd64), as explained in this blog
> post:
> 
> <http://blogs.msdn.com/b/michael_howard/archive/2005/12/06/500629.aspx>

Ah, great point.  I improved the clang codegen to this:

$ cat t.cc 
void *test(long count) {
      return new int[count];
}
$ clang t.cc -S -o - -O3 -mkernel -fomit-frame-pointer -mllvm -show-mc-encoding
	.section	__TEXT,__text,regular,pure_instructions
	.globl	__Z4testl
	.align	4, 0x90
__Z4testl:                              ## @_Z4testl
## BB#0:                                ## %entry
	movl	$4, %ecx                ## encoding: [0xb9,0x04,0x00,0x00,0x00]
	movq	%rdi, %rax              ## encoding: [0x48,0x89,0xf8]
	mulq	%rcx                    ## encoding: [0x48,0xf7,0xe1]
	movq	$-1, %rdi               ## encoding: [0x48,0xc7,0xc7,0xff,0xff,0xff,0xff]
	cmovnoq	%rax, %rdi              ## encoding: [0x48,0x0f,0x41,0xf8]
	jmp	__Znam                  ## TAILCALL
                                        ## encoding: [0xeb,A]
                                        ##   fixup A - offset: 1, value: __Znam-1, kind: FK_PCRel_1
.subsections_via_symbols

This could be further improved by inverting the cmov condition to avoid the first movq, which we'll tackle as a general regalloc improvement.

Thanks for the pointer!

-Chris

Re: operator new[] overflow (PR 19351)

[Richard Guenther]

On Sun, Dec 5, 2010 at 8:56 AM, Chris Lattner <clattner@apple.com> wrote:
>
> On Dec 4, 2010, at 5:22 AM, Florian Weimer wrote:
>
>> * Joe Buck:
>>
>>> It's wasted code if the multiply instruction detects the overflow.
>>> It's true that the cost is small (maybe just one extra instruction
>>> and the same number of tests, maybe one more on architectures where you
>>> have to load a large constant), but it is slightly worse code than what
>>> Chris Lattner showed.
>>
>> It's possible to improve slightly on the LLVM code by using the
>> overflow flag (at least on i386/amd64), as explained in this blog
>> post:
>>
>> <http://blogs.msdn.com/b/michael_howard/archive/2005/12/06/500629.aspx>
>
> Ah, great point.  I improved the clang codegen to this:
>
> $ cat t.cc
> void *test(long count) {
>      return new int[count];
> }
> $ clang t.cc -S -o - -O3 -mkernel -fomit-frame-pointer -mllvm -show-mc-encoding
>        .section        __TEXT,__text,regular,pure_instructions
>        .globl  __Z4testl
>        .align  4, 0x90
> __Z4testl:                              ## @_Z4testl
> ## BB#0:                                ## %entry
>        movl    $4, %ecx                ## encoding: [0xb9,0x04,0x00,0x00,0x00]
>        movq    %rdi, %rax              ## encoding: [0x48,0x89,0xf8]
>        mulq    %rcx                    ## encoding: [0x48,0xf7,0xe1]
>        movq    $-1, %rdi               ## encoding: [0x48,0xc7,0xc7,0xff,0xff,0xff,0xff]
>        cmovnoq %rax, %rdi              ## encoding: [0x48,0x0f,0x41,0xf8]
>        jmp     __Znam                  ## TAILCALL
>                                        ## encoding: [0xeb,A]
>                                        ##   fixup A - offset: 1, value: __Znam-1, kind: FK_PCRel_1
> .subsections_via_symbols
>
> This could be further improved by inverting the cmov condition to avoid the first movq, which we'll tackle as a general regalloc improvement.

I'm curious as on how you represent the overflow checking in your highlevel IL.

Richard.

> Thanks for the pointer!
>
> -Chris
>
>

Re: operator new[] overflow (PR 19351)

[Chris Lattner]

On Dec 5, 2010, at 3:19 AM, Richard Guenther wrote:

>> $ clang t.cc -S -o - -O3 -mkernel -fomit-frame-pointer -mllvm -show-mc-encoding
>>        .section        __TEXT,__text,regular,pure_instructions
>>        .globl  __Z4testl
>>        .align  4, 0x90
>> __Z4testl:                              ## @_Z4testl
>> ## BB#0:                                ## %entry
>>        movl    $4, %ecx                ## encoding: [0xb9,0x04,0x00,0x00,0x00]
>>        movq    %rdi, %rax              ## encoding: [0x48,0x89,0xf8]
>>        mulq    %rcx                    ## encoding: [0x48,0xf7,0xe1]
>>        movq    $-1, %rdi               ## encoding: [0x48,0xc7,0xc7,0xff,0xff,0xff,0xff]
>>        cmovnoq %rax, %rdi              ## encoding: [0x48,0x0f,0x41,0xf8]
>>        jmp     __Znam                  ## TAILCALL
>>                                        ## encoding: [0xeb,A]
>>                                        ##   fixup A - offset: 1, value: __Znam-1, kind: FK_PCRel_1
>> .subsections_via_symbols
>> 
>> This could be further improved by inverting the cmov condition to avoid the first movq, which we'll tackle as a general regalloc improvement.
> 
> I'm curious as on how you represent the overflow checking in your highlevel IL.

The (optimized) generated IR is:

$ clang t.cc -emit-llvm -S -o - -O3
...
define noalias i8* @_Z4testl(i64 %count) ssp {
entry:
  %0 = tail call %0 @llvm.umul.with.overflow.i64(i64 %count, i64 4)
  %1 = extractvalue %0 %0, 1
  %2 = extractvalue %0 %0, 0
  %3 = select i1 %1, i64 -1, i64 %2
  %call = tail call noalias i8* @_Znam(i64 %3)
  ret i8* %call
}

More information on the overflow intrinsics is here:
http://llvm.org/docs/LangRef.html#int_overflow

-Chris

Re: operator new[] overflow (PR 19351)

[Chris Lattner]

On Dec 5, 2010, at 9:49 AM, Chris Lattner wrote:

> 
> On Dec 5, 2010, at 3:19 AM, Richard Guenther wrote:
> 
>>> $ clang t.cc -S -o - -O3 -mkernel -fomit-frame-pointer -mllvm -show-mc-encoding
>>>       .section        __TEXT,__text,regular,pure_instructions
>>>       .globl  __Z4testl
>>>       .align  4, 0x90
>>> __Z4testl:                              ## @_Z4testl
>>> ## BB#0:                                ## %entry
>>>       movl    $4, %ecx                ## encoding: [0xb9,0x04,0x00,0x00,0x00]
>>>       movq    %rdi, %rax              ## encoding: [0x48,0x89,0xf8]
>>>       mulq    %rcx                    ## encoding: [0x48,0xf7,0xe1]
>>>       movq    $-1, %rdi               ## encoding: [0x48,0xc7,0xc7,0xff,0xff,0xff,0xff]
>>>       cmovnoq %rax, %rdi              ## encoding: [0x48,0x0f,0x41,0xf8]
>>>       jmp     __Znam                  ## TAILCALL
>>>                                       ## encoding: [0xeb,A]
>>>                                       ##   fixup A - offset: 1, value: __Znam-1, kind: FK_PCRel_1
>>> .subsections_via_symbols
>>> 
>>> This could be further improved by inverting the cmov condition to avoid the first movq, which we'll tackle as a general regalloc improvement.
>> 
>> I'm curious as on how you represent the overflow checking in your highlevel IL.
> 
> The (optimized) generated IR is:
> 
> $ clang t.cc -emit-llvm -S -o - -O3
> ...
> define noalias i8* @_Z4testl(i64 %count) ssp {
> entry:
>  %0 = tail call %0 @llvm.umul.with.overflow.i64(i64 %count, i64 4)
>  %1 = extractvalue %0 %0, 1
>  %2 = extractvalue %0 %0, 0
>  %3 = select i1 %1, i64 -1, i64 %2
>  %call = tail call noalias i8* @_Znam(i64 %3)
>  ret i8* %call
> }

Sorry, it's a little easier to read with expanded names and types:

define noalias i8* @_Z4testl(i64 %count) ssp {
entry:
  %A = tail call { i64, i1 } @llvm.umul.with.overflow.i64(i64 %count, i64 4)
  %B = extractvalue { i64, i1 } %A, 1
  %C = extractvalue { i64, i1 } %A, 0
  %D = select i1 %B, i64 -1, i64 %C
  %call = tail call noalias i8* @_Znam(i64 %D)
  ret i8* %call
}

-Chris

Re: operator new[] overflow (PR 19351)

[Richard Guenther]

On Sun, Dec 5, 2010 at 6:49 PM, Chris Lattner <clattner@apple.com> wrote:
>
> On Dec 5, 2010, at 3:19 AM, Richard Guenther wrote:
>
>>> $ clang t.cc -S -o - -O3 -mkernel -fomit-frame-pointer -mllvm -show-mc-encoding
>>>        .section        __TEXT,__text,regular,pure_instructions
>>>        .globl  __Z4testl
>>>        .align  4, 0x90
>>> __Z4testl:                              ## @_Z4testl
>>> ## BB#0:                                ## %entry
>>>        movl    $4, %ecx                ## encoding: [0xb9,0x04,0x00,0x00,0x00]
>>>        movq    %rdi, %rax              ## encoding: [0x48,0x89,0xf8]
>>>        mulq    %rcx                    ## encoding: [0x48,0xf7,0xe1]
>>>        movq    $-1, %rdi               ## encoding: [0x48,0xc7,0xc7,0xff,0xff,0xff,0xff]
>>>        cmovnoq %rax, %rdi              ## encoding: [0x48,0x0f,0x41,0xf8]
>>>        jmp     __Znam                  ## TAILCALL
>>>                                        ## encoding: [0xeb,A]
>>>                                        ##   fixup A - offset: 1, value: __Znam-1, kind: FK_PCRel_1
>>> .subsections_via_symbols
>>>
>>> This could be further improved by inverting the cmov condition to avoid the first movq, which we'll tackle as a general regalloc improvement.
>>
>> I'm curious as on how you represent the overflow checking in your highlevel IL.
>
> The (optimized) generated IR is:
>
> $ clang t.cc -emit-llvm -S -o - -O3
> ...
> define noalias i8* @_Z4testl(i64 %count) ssp {
> entry:
>  %0 = tail call %0 @llvm.umul.with.overflow.i64(i64 %count, i64 4)
>  %1 = extractvalue %0 %0, 1
>  %2 = extractvalue %0 %0, 0
>  %3 = select i1 %1, i64 -1, i64 %2
>  %call = tail call noalias i8* @_Znam(i64 %3)
>  ret i8* %call
> }
>
> More information on the overflow intrinsics is here:
> http://llvm.org/docs/LangRef.html#int_overflow

Ah, you're using intrinsics.  I thought of re-using the saturating arithmetic
and types we have, thus basically do

  size = (unsigned sat int) count * 4;

and defer optimal expansion to an optab.  It of course requires saturating
arithmetic emulation for targets that don't provide an expander but would
allow optimal expansion at least.

And it'll unleash all the latent bugs we have with saturating types ...

Richard.

> -Chris

Re: operator new[] overflow (PR 19351)

[Joseph S. Myers]

On Sun, 5 Dec 2010, Richard Guenther wrote:

> Ah, you're using intrinsics.  I thought of re-using the saturating arithmetic
> and types we have, thus basically do
> 
>   size = (unsigned sat int) count * 4;
> 
> and defer optimal expansion to an optab.  It of course requires saturating
> arithmetic emulation for targets that don't provide an expander but would
> allow optimal expansion at least.
> 
> And it'll unleash all the latent bugs we have with saturating types ...

And I thought that defining saturated operations for normal integer types 
would be better than trying to use fixed-point types on targets that don't 
have an ABI for them or where they may not match the types on which you 
want saturated operations.  That is, I think saturated operations would be 
a better internal representation at the GIMPLE level than saturated types 
(this may also apply to lowering existing fixed-point saturated type 
support) and then they could be expanded to various forms of RTL including 
direct saturated instructions, comparisons or overflow flags checks.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: operator new[] overflow (PR 19351)

[Richard Guenther]

On Sun, Dec 5, 2010 at 10:21 PM, Joseph S. Myers
<joseph@codesourcery.com> wrote:
> On Sun, 5 Dec 2010, Richard Guenther wrote:
>
>> Ah, you're using intrinsics.  I thought of re-using the saturating arithmetic
>> and types we have, thus basically do
>>
>>   size = (unsigned sat int) count * 4;
>>
>> and defer optimal expansion to an optab.  It of course requires saturating
>> arithmetic emulation for targets that don't provide an expander but would
>> allow optimal expansion at least.
>>
>> And it'll unleash all the latent bugs we have with saturating types ...
>
> And I thought that defining saturated operations for normal integer types
> would be better than trying to use fixed-point types on targets that don't
> have an ABI for them or where they may not match the types on which you
> want saturated operations.  That is, I think saturated operations would be
> a better internal representation at the GIMPLE level than saturated types
> (this may also apply to lowering existing fixed-point saturated type
> support) and then they could be expanded to various forms of RTL including
> direct saturated instructions, comparisons or overflow flags checks.

True - but it's not how saturated operations are implemented in the
middle-end right now.  Eventually I will finish transitioning our undefined
overflow scheme to that for 4.7 though ...

Richard.

> --
> Joseph S. Myers
> joseph@codesourcery.com
>

Re: operator new[] overflow (PR 19351)

[Florian Weimer]

* Gabriel Dos Reis:

>>> Still, it's certainly an improvement on the current
>>> situation and the cost is negligible compared to the call to the
>>> allocator.  Since it's a security issue, some form of the patch should
>>> go in.
>>
>> Well, should I resubmit, with the fix for the problem building
>> size_t(-1)?
>
> I think that would help.

I got it to work, finally:

  <http://gcc.gnu.org/ml/gcc-patches/2011-01/msg01593.html>

I hope it helps fixing this issue for good.