[Bug uprobes/9826] New: gdb/uprobes deathmatch goes to OT

[Bug uprobes/9826] New: gdb/uprobes deathmatch goes to OT

[jkenisto at us dot ibm dot com]

In bugzilla #7082, Comment #27, Marl Wielaard said:

> I ran the stap script without -c:
> $ stap -ve 'probe process("bash").function("*").return {}' -w
> 
> Then in another window:
> (gdb) run
> Starting program: /bin/bash 
> 
> Program received signal SIGTRAP, Trace/breakpoint trap.
> 0x0000000000424b70 in open () at /usr/include/bits/fcntl2.h:54
> 54	      return __open_alias (__path, __oflag, __va_arg_pack ());
> (gdb) bt
> #0  0x0000000000424b70 in open () at /usr/include/bits/fcntl2.h:54
> #1  check_dev_tty () at general.c:381
> Cannot access memory at address 0xffffe518
> 
> > In any case, do you see the same sort of failure if you turn the return probe
> > into an entry probe (a al #7092)?
> 
> That also fails, although slightly differently:
> $ stap -ve 'probe process("bash").function("main") {}' -w
> 
> And in GDB:
> (gdb) run
> Starting program: /bin/bash 
> 
> Program terminated with signal SIGILL, Illegal instruction.
> The program no longer exists.
> You can't do that without a process to debug.
> 
> In both cases 2.6.27.12-170.2.5.fc10.x86_64 under xen.

The basic problem here is that gdb and stap/uprobes are both trying
to set breakpoints at the same spot -- e.g., at the entry to main().

If gdb sets the breakpoint first, uprobes will notice it and refuse
to set a probepoint there (and both uprobes and the stap-generated
module will report this via printk()).

But if uprobes sets the breakpoint first (as in this case), gdb 
doesn't notice, and Bad Things ensue when the breakpoint is hit.
Uprobes tries to single-step the original instruction out of line,
and at the same time, gdb tries to single-step the int3 via opcode
replacement.  Typically, a SIGSEGV ensues... but that's not the worst
of it.

The uprobes code that enables it to persevere when an
unexpected-but-harmless signal such as SIGPROF or SIGALRM arrives in
the middle of SSOL -- see uprobe_delay_signal() -- also causes uprobes
to set aside the SIGSEGVs that gdb (or maybe utrace) keeps sending its
way -- tens of thousands per second.  ("When I'm done single-stepping,
I'll pass that SIGSEGV along...  and that one... and that one...")
Unless/until you kill -9 the probed process, gdb and stap hang,
and uprobes eats up memory with delayed_signal objects.

Repeat-by: Use stap to probe main() in any program -- even hello.c --
and then try to run an instance of that program in gdb.

Workaround: Don't probe the same instructions with both gdb and stap.
If you do, you can get out of it by killing the probed process with
kill -9.  (You can't kill gdb or the stap session while the probed
process is hung like this.)

Solution: (1) Teach gdb to check for an existing breakpoint instruction
before trying to insert one.  (2) Teach uprobes to give up eventually,
when appropriate.  I'm pursuing (2).

-- 
           Summary: gdb/uprobes deathmatch goes to OT
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: uprobes
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: jkenisto at us dot ibm dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=9826

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/9826] gdb/uprobes deathmatch goes to OT

[mhiramat at redhat dot com]

------- Additional Comments From mhiramat at redhat dot com  2009-02-06 22:14 -------
Can gdb use uprobe for probing target process?
I think if the kernel supports a 'standard' method of setup break point, gdb
doesn't need to tweak target code manually.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=9826

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug uprobes/9826] gdb/uprobes deathmatch goes to OT

[Jim Keniston]

On Fri, 2009-02-06 at 22:14 +0000, mhiramat at redhat dot com wrote:
> ------- Additional Comments From mhiramat at redhat dot com  2009-02-06 22:14 -------
> Can gdb use uprobe for probing target process?
> I think if the kernel supports a 'standard' method of setup break point, gdb
> doesn't need to tweak target code manually.
> 

[I'm replying outside bugzilla, since topic is much broader than the
subject of #9826.]

Certainly there's a shared vision of a user-space-breakpoint service
that could be used by kernel-side services like uprobes and by a
system-call API like ptrace or its successor.  A key feature of this
service would be to support probing of multithreaded apps with no probe
misses, as provided by uprobes and kprobes.

The ubp API I proposed here
https://www.redhat.com/archives/utrace-devel/2008-November/msg00020.h
is a first stab at such a service.  I actually coded up an x86
implementation of it -- and sent a copy to roland, cmoller, and some
Bangalore IBMers Nov. 21 -- but I never got around to testing it.  (I'll
forward you a copy, Masami.)

Roland later suggested that we factor instruction analysis out of ubp,
and also expand ubp to accommodate architectures that don't do
single-stepping per se.

As for building a system-call API atop the current uprobes... I don't
know of any support for pursuing that.

Jim

[Bug uprobes/9826] gdb/uprobes deathmatch goes to OT

[jkenisto at us dot ibm dot com]

------- Additional Comments From jkenisto at us dot ibm dot com  2009-02-06 23:33 -------
(In reply to comment #1)
> Can gdb use uprobe for probing target process?
...

See http://sources.redhat.com/ml/systemtap/2009-q1/msg00396.html for my reply to
this.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=9826

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug uprobes/9826] gdb/uprobes deathmatch goes to OT

[Chris Moller]

[-- Attachment #1: Type: text/plain, Size: 1945 bytes --]



Jim Keniston wrote:
> On Fri, 2009-02-06 at 22:14 +0000, mhiramat at redhat dot com wrote:
>   
>> ------- Additional Comments From mhiramat at redhat dot com  2009-02-06 22:14 -------
>> Can gdb use uprobe for probing target process?
>> I think if the kernel supports a 'standard' method of setup break point, gdb
>> doesn't need to tweak target code manually.
>>
>>     
>
> [I'm replying outside bugzilla, since topic is much broader than the
> subject of #9826.]
>
> Certainly there's a shared vision of a user-space-breakpoint service
> that could be used by kernel-side services like uprobes and by a
> system-call API like ptrace or its successor.  A key feature of this
> service would be to support probing of multithreaded apps with no probe
> misses, as provided by uprobes and kprobes.
>
> The ubp API I proposed here
> https://www.redhat.com/archives/utrace-devel/2008-November/msg00020.h
> is a first stab at such a service.  I actually coded up an x86
> implementation of it -- and sent a copy to roland, cmoller, and some
>   

I'm trying to incorporate that copy into the froggy module and access it
through the froggy user-space API   That's part of a lot of other stuff,
so it's not happening over night, but I expect I'll get something that
kinda limps along in a month or so.

> Bangalore IBMers Nov. 21 -- but I never got around to testing it.  (I'll
> forward you a copy, Masami.)
>
> Roland later suggested that we factor instruction analysis out of ubp,
> and also expand ubp to accommodate architectures that don't do
> single-stepping per se.
>
> As for building a system-call API atop the current uprobes... I don't
> know of any support for pursuing that.
>
> Jim
>
>
>
>   

-- 
Chris Moller

  I know that you believe you understand what you think I said, but
  I'm not sure you realize that what you heard is not what I meant.
      -- Robert McCloskey



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[Bug uprobes/9826] gdb/uprobes deathmatch goes to OT

[ananth at in dot ibm dot com]

------- Additional Comments From ananth at in dot ibm dot com  2009-02-07 11:30 -------
I guess in the short term, Roland's utrace-ptrace mutual exclusion should help.
If the Fedora kernels haven't enabled that option, maybe they should, till the
details of ptrace (and gdb) and utrace/uprobes interaction are worked out.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=9826

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug uprobes/9826] gdb/uprobes deathmatch goes to OT

[srikar at linux dot vnet dot ibm dot com]

------- Additional Comments From srikar at linux dot vnet dot ibm dot com  2009-02-09 10:58 -------
Update from Jim Keniston:
-------------------------

For the purposes of fixing this particular bug, I think it's sufficient 
to modify the code that calls uprobe_delay_signal() to refuse to delay the 
signal if it's a SIGSEGV -- something like this:  

        if (signal_action == UTRACE_SIGNAL_DELIVER && utask->active_probe && 
                                        info->si_signo != SSTEP_SIGNAL) { 
                if (info->si_signo == SIGSEGV) {
                        /*
                         * Give up and let the SIGSEGV be delivered.
                         * uprobe_report_exit() will unlock the semaphores, etc.
                         */
                        return UTRACE_SIGNAL_DELIVER | UTRACE_RESUME;
                } else {                      
                        uprobe_delay_signal(utask, info);
                        uprobe_decref_process(uproc);
                        return UTRACE_SIGNAL_IGN | UTRACE_SINGLESTEP;
                }
        }
                                                                               
                                                         
A more robust solution might enforce a limit on the number of times 
uprobe_report_signal() can be called -- for a particular task, while
attempting a particular single-step operation -- before you give up 
and abort the single-step.  But again, be more tolerant of signals like 
SIGPROF and SIGALRM.
                                                                               
                                                         
BTW, in this particular case, another indication that things are fubar 
is that regs->ip no longer points to the right SSOL slot.  (gdb has 
decremented regs->ip, thinking that it's backing up to the beginning 
of the probed instruction.)  But that indication doesn't work on 
architectures where the breakpoint instruction doesn't advance the IP.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|systemtap at sources dot    |srikar at linux dot vnet dot
                   |redhat dot com              |ibm dot com
             Status|NEW                         |ASSIGNED


http://sourceware.org/bugzilla/show_bug.cgi?id=9826

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: [Bug uprobes/9826] gdb/uprobes deathmatch goes to OT

[Mark Wielaard]

Hi Jim,

On Fri, 2009-02-06 at 15:20 -0800, Jim Keniston wrote:
> On Fri, 2009-02-06 at 22:14 +0000, mhiramat at redhat dot com wrote:
> > ------- Additional Comments From mhiramat at redhat dot com  2009-02-06 22:14 -------
> > Can gdb use uprobe for probing target process?
> > I think if the kernel supports a 'standard' method of setup break point, gdb
> > doesn't need to tweak target code manually.
> > 
> [I'm replying outside bugzilla, since topic is much broader than the
> subject of #9826.]

Thanks for taking this to the list.
And sorry for confusing matters on bug #7082.
To quote from that one to get some context:

> In bugzilla #7082, Comment #27, Mark Wielaard said:
> > > In any case, do you see the same sort of failure if you turn the return probe
> > > into an entry probe (a al #7092)?
> > 
> > That also fails, although slightly differently:
> > $ stap -ve 'probe process("bash").function("main") {}' -w
> > 
> > And in GDB:
> > (gdb) run
> > Starting program: /bin/bash 
> > 
> > Program terminated with signal SIGILL, Illegal instruction.
> > The program no longer exists.
> > You can't do that without a process to debug.
> > 
> > In both cases 2.6.27.12-170.2.5.fc10.x86_64 under xen.
> 
> The basic problem here is that gdb and stap/uprobes are both trying
> to set breakpoints at the same spot -- e.g., at the entry to main().

Aha, OK. So it wasn't my intention to highlight that fact.
And I actually didn't know that would happen.

I was just running the program being traced under gdb to show how/why it
crashed (with a SIGILL, SIGSEGV or SIGTRAP) when tracing with uprobes.
Now I see gdb might be lying to me. doh.

What method do you use to inspect a user space probe going wrong if you
cannot rely on gdb?

Thanks,

Mark