function probe oddities

function probe oddities

[Stone, Joshua I]

I've recently noticed that it's possible to have function probe that
don't behave quite like one would expect.  The problem is when you have
a loop at the very beginning of a function, such that there aren't any
prior instructions that aren't part of the loop.  You then get a probe
that fires on every iteration of the loop, instead of only at the
entrance to the function as desired.

Consider this trivial example:

void simple()
{
    do
        work();
    while (test());
}

00000000 <simple>:
   0:   e8 0a 00 00 00          call   f <work>
   5:   e8 06 00 00 00          call   10 <test>
   a:   85 c0                   test   %eax,%eax
   c:   75 f2                   jne    0 <simple>
   e:   c3                      ret


There isn't anywhere that you could place a probe that will only fire at
the beginning of the function!  This kind of thing is probably a rare
occurrence, but I have seen it in real code.  I'm not sure that we can
do anything about it, but this limitation should at least be noted.

Also affected are kretprobes, which may end up registering the kretprobe
instance many times.

On a related note, I have noticed that function() and function().return
probes are not always placed at the same address.  I would guess that
this is because the former needs to be after the prologue, but the
latter doesn't care about the prologue.  However, since two aggregate
probes would be faster than taking two consecutive debug traps, it would
be nice if these used the same address.


My 'real-world' scenario that triggered this investigation had a
function with a small prologue followed immediately by a loop body.  I
was trying to trace the call sequence using thread_indent().  I ended up
with what looked like a whole lot of function entrances with only one
function exit.  It turns out that the call-probe (post-prologue) was
firing every time through the loop, but the return-probe (pre-prologue)
only fired once.

Given that we may not be able to fix the looping problem, I think I
would prefer that the return probes carried the same weird behavior.  If
the return probe used the same address as the call probe, then at least
I could get a 1:1 mapping between call/return probes...


Josh

Re: function probe oddities

[Jim Keniston]

On Tue, 2006-07-11 at 14:52, Stone, Joshua I wrote:
> I've recently noticed that it's possible to have function probe that
> don't behave quite like one would expect.  The problem is when you have
> a loop at the very beginning of a function, such that there aren't any
> prior instructions that aren't part of the loop.  You then get a probe
> that fires on every iteration of the loop, instead of only at the
> entrance to the function as desired.
> 
> Consider this trivial example:
> 
> void simple()
> {
>     do
>         work();
>     while (test());
> }
> 
> 00000000 <simple>:
>    0:   e8 0a 00 00 00          call   f <work>
>    5:   e8 06 00 00 00          call   10 <test>
>    a:   85 c0                   test   %eax,%eax
>    c:   75 f2                   jne    0 <simple>
>    e:   c3                      ret
> 

So weird, yet so innocent.  (The problem, not Josh. :-))

> 
> There isn't anywhere that you could place a probe that will only fire at
> the beginning of the function!  This kind of thing is probably a rare
> occurrence, but I have seen it in real code.  I'm not sure that we can
> do anything about it, but this limitation should at least be noted.

Yes, a patch to Documentation/kprobes.txt is warranted.  I'll do it when
I get back from vacation, if nobody beats me to it.

> 
> Also affected are kretprobes, which may end up registering the kretprobe
> instance many times.

Yes, indeed.

> 
> On a related note, I have noticed that function() and function().return
> probes are not always placed at the same address.  I would guess that
> this is because the former needs to be after the prologue, but the
> latter doesn't care about the prologue.

Kretprobes wants kp.addr to be the first instruction of the function, so
it can be sure of where to find the return address (as specified by the
ABI).  There's actually no reason you couldn't specify a later
instruction, so long as the return address is still where kretprobes
looks for it.  That wouldn't help in this case, though.

> However, since two aggregate
> probes would be faster than taking two consecutive debug traps, it would
> be nice if these used the same address.

Nice, but not a big concern.

> My 'real-world' scenario that triggered this investigation had a
> function with a small prologue followed immediately by a loop body.  I
> was trying to trace the call sequence using thread_indent().  I ended up
> with what looked like a whole lot of function entrances with only one
> function exit.  It turns out that the call-probe (post-prologue) was
> firing every time through the loop, but the return-probe (pre-prologue)
> only fired once.
> 
> Given that we may not be able to fix the looping problem, I think I
> would prefer that the return probes carried the same weird behavior.  If
> the return probe used the same address as the call probe, then at least
> I could get a 1:1 mapping between call/return probes...

As previously mentioned, adjusting kretprobes would be problematic.

The gcc folks may come to our rescue, though -- e.g., by helping
SystemTap notice such situations.  Back when SystemTap was having
trouble locating the end of the prologue for some functions, I suggested
training SystemTap in each architecture's ABI, so it could find the args
right at function entry.  The preferred solution, as I recall, was to
enhance the dwarf info.

It's an interesting problem, all right.

> 
> 
> Josh

Jim

Re: function probe oddities

[Roland McGrath]

When there is no prologue, there is indeed nothing really to do.  In the
arbitrarily far-off future in the context of arbitrarily vastly more
convenient access to arbitrarily extensive machine features, a trap there
could get the location before last branch from the CPU in magic ways and so
distinguish a call from an iteration.  But short of that, it's hard to see.

Probably more common are situations where there is a little bit of prologue
before the loop.  For those, we could do better by making entry probes
start at the beginning rather than the end of the prologue.  The main
reason we made the translator go to the trouble of finding the end of the
prologue is that originally GCC's DWARF location information for the
parameters was often unreliable at PCs inside the prologue.  We believe
that the latest GCC versions we're now using to build new kernels have
thoroughly fixed that problem.  I even think the RHEL4 version of GCC 3.4
has had enough of the same fixed backported that it doesn't have these
problems.  But that was not the case when we got systemtap entry probes
with target variable access working originally.  

Since there was so much effort put into it back then, I think we have been
understandably loathe to mess with the status quo.  It's the conservative
thing to keep it, especially if there is any desire to widen the set of
compilers for building a kernel that systemtap can work with.  

If you're up for some experimentation, it might be worthwhile to add a
switch to disable the prologue detection.  Then try all the target variable
access you can, and see if it all still works as it did before.

There are reasons other than old GCC bugs that function entry breakpoints
going after prologue code is desireable.  For traditional debugger
breakpoints, that's the point at which stepping will do something
meaningful.  When you want a breakpoint for the purpose of changing some
program state, be it a traditional debugger or a write-capable guru probe,
then before the prologue completes you might not be able to write into
parameter or local variable locations and have your changes affect the
source level variables the way you want them to.  Those concerns are
probably not immediately relevant for systemtap.


Thanks,
Roland