[Bug target/16601] New: [3.3 regression] miscompilation of automatic dynamic arrays in crypto/IPsec subsystems of the Linux kernel

[Bug target/16601] New: [3.3 regression] miscompilation of automatic dynamic arrays in crypto/IPsec subsystems of the Linux kernel

[debian-gcc at lists dot debian dot org]

[forwarded from http://bugs.debian.org/259887]

regression from 3.2.3, rechecked with 3.3.4, fixed in 3.4.

    Matthias

Herbert Xu writes:

With the options -O2 -mpreferred-stack-boundary=2 (default flags for kernel
compilation, gcc 3.3.4 is miscompiling automatic dynamic arrays.  Unfortunately
both are used in the crypto/IPsec subsystems of the Linux kernel. 
 
Here is a sample program: 

#include <string.h> 
 
int bar(char *s); 
 
int foo(char *s, int len, int x) 
{ 
        char buf[x ? len : 0]; 
 
        if (x) { 
                memcpy(buf, s, len); 
                s = buf; 
        } 
 
        return bar(s); 
} 
 
With gcc 3.3.4, this produces: 

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
        .file   "b.c" 
        .text 
        .p2align 4,,15 
.globl foo 
        .type   foo, @function 
foo: 
        pushl   %ebp 
        xorl    %eax, %eax 
        movl    %esp, %ebp 
        subl    $24, %esp 
        movl    16(%ebp), %ecx 
        movl    %edi, -4(%ebp) 
        movl    12(%ebp), %edx 
        movl    %esp, %edi 
        movl    %ebx, -12(%ebp) 
        movl    %esi, -8(%ebp) 
        decl    %edx 
        movl    8(%ebp), %esi 
        testl   %ecx, %ecx 
        setne   %al 
        decl    %eax 
        orl     %eax, %edx 
        addl    $19, %edx 
        andl    $-4, %edx 
--------------------------------------------------------------------- 
        subl    %edx, %esp 
        leal    27(%esp), %ebx 
        andl    $-16, %ebx 
 
Note the offset 27.  The same program when compiled with gcc 3.2.3 
produces similar output but it uses an offset of 15. 
 
Suppose that len = 16, x != 0, and %esp & 15 = 8 before the subl. 
 
That means %edx = (15 + 19) & ~3 = 32.  So %esp & 15 is still 8 
after the subtraction.  That is, %esp = 16x + 8.  Hence 
%ebx = (%esp + 27) & ~15 = (16x + 35) & ~15 = 16x + 32 = %esp + 24. 
 
Therefore buf will only contain 8 bytes of space instead of 16 
bytes. 
--------------------------------------------------------------------- 
        testl   %ecx, %ecx 
        jne     .L5 
.L4: 
        movl    %esi, (%esp) 
        call    bar 
        movl    %edi, %esp 
        movl    -12(%ebp), %ebx 
        movl    -8(%ebp), %esi 
        movl    -4(%ebp), %edi 
        movl    %ebp, %esp 
        popl    %ebp 
        ret 
        .p2align 4,,7 
.L5: 
        movl    12(%ebp), %eax 
        movl    %esi, 4(%esp) 
        movl    %ebx, %esi 
        movl    %eax, 8(%esp) 
        movl    %ebx, (%esp) 
        call    memcpy 
        jmp     .L4 
        .size   foo, .-foo 
        .section        .note.GNU-stack,"",@progbits 
        .ident  "GCC: (GNU) 3.3.4 (Debian 1:3.3.4-3)" 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
 
Since this bug can lead to remotely triggered crashes and possibly 
exploits I'm rating it as critical.

-- 
           Summary: [3.3 regression] miscompilation of automatic dynamic
                    arrays in crypto/IPsec subsystems of the Linux kernel
           Product: gcc
           Version: 3.3.4
            Status: UNCONFIRMED
          Severity: critical
          Priority: P2
         Component: target
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: debian-gcc at lists dot debian dot org
                CC: gcc-bugs at gcc dot gnu dot org,gdr at gcc dot gnu dot
                    org,herbert at gondor dot apana dot org dot au
 GCC build triplet: ix86-linux
  GCC host triplet: ix86-linux
GCC target triplet: ix86-linux


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16601

[Bug target/16601] [3.3 regression] miscompilation of automatic dynamic arrays in crypto/IPsec subsystems of the Linux kernel

[debian-gcc at lists dot debian dot org]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |wrong-code
      Known to fail|                            |3.3.4
      Known to work|                            |3.2.3 3.4.1


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16601

[Bug target/16601] [3.3 regression] miscompilation of automatic dynamic arrays in crypto/IPsec subsystems of the Linux kernel

[pinskia at gcc dot gnu dot org]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |3.3.5


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16601

[Bug target/16601] [3.3 regression] miscompilation of automatic dynamic arrays in crypto/IPsec subsystems of the Linux kernel

[herbert at gondor dot apana dot org dot au]

------- Additional Comments From herbert at gondor dot apana dot org dot au  2004-07-17 19:30 -------
I'm sorry but I got this wrong.  The compiler is allocating 12 bytes of headroom
unconditionally so it is actually safe.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16601

[Bug target/16601] [3.3 regression] miscompilation of automatic dynamic arrays in crypto/IPsec subsystems of the Linux kernel

[falk at debian dot org]

------- Additional Comments From falk at debian dot org  2004-08-24 21:31 -------
(In reply to comment #1)
> I'm sorry but I got this wrong.  The compiler is allocating 12 bytes of headroom
> unconditionally so it is actually safe.

Okay, let's close this then.

As a side note, the array size must never evaluate to zero, or you'll invoke the
crashes and exploits you mentioned.


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |INVALID


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16601