Hi,

This patch makes the TestLibFunctionStepFrame PASS (but see below) and
makes nexting over shared library function calls (through plt entries)
work on x86.

To recap the problem. When there is no debug/unwind_frame info libunwind
needs to guess how to do a step. The final fallback on x86 is using the
frame pointer (ESB). There are two problems with this. First code
compiled with -fomit-frame-pointer doesn't have frame pointers, so then
we are chasing a possibly ghost frame. Second when stepping onto a plt
entry there is no real frame setup, the plt entry is kind of a jump
point to the real function to be called (and facilitates invoking the
dynamic linker of course to resolve the "jump slot" first). This means
that if you try to follow the frame pointer you are actually following
into the frame above the "plt frame". This confuses the SteppingEngine
terribly. This isn't a direct problem on x86_64 (at least for the
SteppingEngine at this time) because libunwind never tries to do any
frame pointer tricks there. So on x86_64 you currently cannot unwind,
which is better than getting the wrong outer frame. (On x86_64 nexting
works, but TestLibFunctionStepFrame doesn't PASS.)

There are a couple of properties of x86 plt entries that help with
resolving this issue. 1) They only occur as inner frames (think of them
as tail-calls). 2) They are always invoked with a CALL instruction (as
if it was a real function call). 3) If they are resolved they are just
one JMP instruction, with the return address on the stack, otherwise
they are a JMP, two PUSHs (selector and got table address) and a final
JUMP (into ld). The patch tries to recognize this situation.

Because we don't have much info to go on, we don't have any of the elf
section info inside libunwind, it is a bit heuristic at the moment. It
probes the stack for the pattern mentioned above, pulls out the return
address and checks if it comes from a CALL instruction, then adjusts the
stack and ip addresses for the cursor accordingly. I don't know if
upstream will like this solution (it contains one probe into an address
to see what the instruction there is, that could possibly not exist,
which is fine for us since that is just an error access and we fall
back, but when using libunwind in-process this could possibly be fatal
with a really bad address).

One change was made to the test. TestLibFunctionStepFrame used to step
through all of the dynamic loader and made sure all possible
instructions inside had a good backtrace. But there is one point that
seems to have bad unwind info (and of dl_fixup, the last ret
instruction). I filed #5917 to investigate this more. The test has been
adjusted to just check the first few instructions of the actual PLT and
ld and/or actual function entry.
                                      
2008-03-12  Mark Wielaard  <mwielaard@redhat.com>

    * TestLibFunctionStepFrame.java: Mark only unresolved on x86_64
    and ppc. Only check first 24 steps (bug #5917). Tighter check for
    main and foo order.

2008-03-12  Mark Wielaard  <mwielaard@redhat.com>

    * Frame.java (toString): New method.

2007-03-12  Mark Wielaard  <mwielaard@redhat.com>

        * src/x86/Gstep.c (is_call_instr_at): New function.
        (init_stack_based_ret): New function.
        (unw_step): Try stack based unwind with call instr, before
        fallback to frame pointer.

Committed,

Mark