Re: [x32] PATCH: Remove ix86_promote_function_mode

["H. Peter Anvin" <hpa@zytor.com>]

On 06/20/2011 03:49 PM, Richard Henderson wrote:
>>
>> As I have already stated, if we *cannot* require pointers to be
>> zero-extended on entry to the kernel, we're going to have to have
>> special entry points for all the x32 system calls except the ones that
>> don't take pointers.
> 
> If it's a security concern, surely you have to do it in the kernel
> anyway, lest someone call into the kernel via their own assembly
> rather than something controlled by the compiler...
> 

That was the point... right now we rely on the ABI to not have any
invalid representations (except, as far as I know, on s390).  This means
any arbitrary register image presented to the kernel will be a set of
valid C objects; we then accept or reject them as being semantically
valid using normal C code in the kernel.

The issue occurs when the kernel can be entered with something in the
register that is invalid according to the calling convention, and not
have it rejected.

The current x86-64 ABI rules, for example, imply that if
%rdi = 0x3fb8c9119537d37d and the type of the first argument is
uint32_t, that is a valid argument with the value 0x9537d37d.  The extra
upper bits are ignored, and so no security issue arises.

The issue with requiring the upper bits to be normalized occurs with
code like:

static const long foo_table[10] = { ... };

long sys_foo(unsigned int bar)
{
	if (bar >= 10)
		return -EINVAL;

	return foo_table[bar];
}


If the upper bits are required to be zero, gcc could validly translate
that to:

sys_foo:
	cmpl	$10, %edi
	jae	.L1

	movq	foo_table(,%rdi,3), %rax
	retq
.L1:
	movq	$-EINVAL, %rax
	retq

Enter this function with a non-normalized %rdi and you have a security
hole even though the C is perfectly fine.

	-hpa