Help needed to modify prologue/epilogue expand methods...

Help needed to modify prologue/epilogue expand methods...

[Vaibhav Shrimali]

i386.c : ix86_expand_prologue()

/*************************************************************************************************/
x = cfun->machine->force_align_arg_pointer;
x = gen_frame_mem (Pmode, plus_constant (x, -8)) xor hard_frame_pointer_rtx;
insn = emit_insn (gen_push (x));
RTX_FRAME_RELATED_P (insn) = 1;
/*************************************************************************************************/

I am trying to push a copy of return address just below the return
address and xor it with the "frame pointer". I wanted to know if the
above code is correct for pushing the return address and encrypt it
with the frame pointer.
Now, in the epilogue I want to decrypt this copy of return address
with the pushed frame pointer and verify if the two instances
(original and decrypted) of the return address are same. This is the
first time I am working on an approach that requires to make
modifications to the ix86_expand_prologue/epilogue methods of the
compiler. I also wanted to know how to read the value of encrypted
return address in the ix86_expand_epilogue method, since then I need
to decrypt it and compare it with the original return address pushed
on the stack.

I have posted similar requests on other mailing lists but got no reply. Please,
this is critical for my research and i need help from the community.
Please provide
me with any information on the matter that you can.

Thank You.
-- 
Vaibhav Shrimali
BTech(ICT), DA-IICT
Gandhinagar, INDIA

Re: Help needed to modify prologue/epilogue expand methods...

[Ian Lance Taylor]

Vaibhav Shrimali <vaibhav.shrimali@gmail.com> writes:

> i386.c : ix86_expand_prologue()
>
> /*************************************************************************************************/
> x = cfun->machine->force_align_arg_pointer;
> x = gen_frame_mem (Pmode, plus_constant (x, -8)) xor hard_frame_pointer_rtx;
> insn = emit_insn (gen_push (x));
> RTX_FRAME_RELATED_P (insn) = 1;
> /*************************************************************************************************/
>
> I am trying to push a copy of return address just below the return
> address and xor it with the "frame pointer". I wanted to know if the
> above code is correct for pushing the return address and encrypt it
> with the frame pointer.
> Now, in the epilogue I want to decrypt this copy of return address
> with the pushed frame pointer and verify if the two instances
> (original and decrypted) of the return address are same. This is the
> first time I am working on an approach that requires to make
> modifications to the ix86_expand_prologue/epilogue methods of the
> compiler. I also wanted to know how to read the value of encrypted
> return address in the ix86_expand_epilogue method, since then I need
> to decrypt it and compare it with the original return address pushed
> on the stack.

I don't know what you mean by the "xor" above, but otherwise that
looks more or less OK.  You will also have to account for the extra
word in the frame, in struct ix86_frame and ix86_compute_frame_layout.

I recommend that you take a look at the implementation of
-fstack-protector.  I think that is a better approach to this kind of
thing.

Ian