* Segmentation Fault: Task died at uprobe probepoint @ 2010-06-14 9:18 Gaurav Dhiman 2010-06-14 12:55 ` Srikar Dronamraju 2010-06-14 21:06 ` Frank Ch. Eigler 0 siblings, 2 replies; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-14 9:18 UTC (permalink / raw) To: systemtap Hi All, I have recently installed SystemTap on my Ubuntu 9.04 based system. I have Linux kernel 2.6.29.1 with utrace support (I used the utrace patch from http://people.redhat.com/roland/utrace/old/2.6.29/) . I wrote a very simple program to test out the userspace probe functionality. Here is the stap script I wrote: probe process("/home/gdhiman/tests/uprobe-test").begin { printf("uprobe-test begin!!\n") } probe process("/home/gdhiman/tests/uprobe-test").function("test_func") { printf("test_func called!\n"); log($$parms); } The uprobe-test code is as follows: int test_func(int i) { printf("%d\n", i); return j; } int main() { int i; i = test_func(5); return 0; } Then I ran the script as root using stap -v test.stp; it is compiled and inserted successfully. I see the following message if I do dmesg: [ 1694.148460] stap_9736f2321b446f9fc03126194f89aac2_1643: systemtap: 1.2/0.131, base: ffffffffa0010000, memory: 136data/29text/10ctx/13net/34alloc kb, probes: 2 However, when i actually run uprobe-test, although both the probes fire successfully, the task generates a segmentation fault and dies on the 'test_func' probe. I see the following message on doing dmesg: [ 1709.650827] Task died at uprobe probepoint: pid/tgid = 3934/3934, probepoint = 0x400517 I am not sure why is this happening. Is there something wrong in the patch I used, or am I making some stupid mistake in programming the probes? Any insights into what might be happening and how to resolve the issue would be really helpful. Thanks in advance, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-14 9:18 Segmentation Fault: Task died at uprobe probepoint Gaurav Dhiman @ 2010-06-14 12:55 ` Srikar Dronamraju [not found] ` <AANLkTilvvj9hdiaa9VWjO0j-jK02LMweg5cAEBgoLyRK@mail.gmail.com> 2010-06-14 21:06 ` Frank Ch. Eigler 1 sibling, 1 reply; 19+ messages in thread From: Srikar Dronamraju @ 2010-06-14 12:55 UTC (permalink / raw) To: Gaurav Dhiman; +Cc: systemtap Hi, > > probe process("/home/gdhiman/tests/uprobe-test").begin > { > printf("uprobe-test begin!!\n") > } > > probe process("/home/gdhiman/tests/uprobe-test").function("test_func") > { > printf("test_func called!\n"); > log($$parms); > } > > The uprobe-test code is as follows: > > int test_func(int i) > { > printf("%d\n", i); > return j; ^^^^^ Guess you meant "return i;" > } > > int main() > { > int i; > i = test_func(5); > return 0; > } > > > Then I ran the script as root using stap -v test.stp; it is compiled > and inserted successfully. I see the following message if I do dmesg: > [ 1694.148460] stap_9736f2321b446f9fc03126194f89aac2_1643: systemtap: > 1.2/0.131, base: ffffffffa0010000, memory > 136data/29text/10ctx/13net/34alloc kb, probes: 2 Looks to me that you are using a pretty recent systemtap with a pretty old elfutils. Is is possible to try building systemtap with a later version of elfutils? > However, when i actually run uprobe-test, although both the probes > fire successfully, the task generates a segmentation fault and dies on > the 'test_func' probe. I see the following message on doing dmesg: > [ 1709.650827] Task died at uprobe probepoint: pid/tgid = 3934/3934, > probepoint = 0x400517 This means that the process died before the uprobe operation completed. i.e either it was still running the handler or singlestep operation hadnt occurred. - Did you try using the core with gdb? Does address in bt and disassemble output of test_func show the same address? I am not able to see this problem. (I am running on a different kernel) -- Regards Srikar ^ permalink raw reply [flat|nested] 19+ messages in thread
[parent not found: <AANLkTilvvj9hdiaa9VWjO0j-jK02LMweg5cAEBgoLyRK@mail.gmail.com>]
* Re: Segmentation Fault: Task died at uprobe probepoint [not found] ` <AANLkTilvvj9hdiaa9VWjO0j-jK02LMweg5cAEBgoLyRK@mail.gmail.com> @ 2010-06-15 9:11 ` Gaurav Dhiman 2010-06-15 17:46 ` Mark Wielaard 2010-06-15 18:17 ` Srikar Dronamraju 1 sibling, 1 reply; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-15 9:11 UTC (permalink / raw) To: Srikar Dronamraju, systemtap On Mon, Jun 14, 2010 at 12:00 AM, Srikar Dronamraju <srikar@linux.vnet.ibm.com> wrote: >> The uprobe-test code is as follows: >> >> int test_func(int i) >> { >> printf("%d\n", i); >> return j; > ^^^^^ > Guess you meant "return i;" Yes, I meant 'i' :) >> Then I ran the script as root using stap -v test.stp; it is compiled >> and inserted successfully. I see the following message if I do dmesg: >> [ 1694.148460] stap_9736f2321b446f9fc03126194f89aac2_1643: systemtap: >> 1.2/0.131, base: ffffffffa0010000, memory >> 136data/29text/10ctx/13net/34alloc kb, probes: 2 > > Looks to me that you are using a pretty recent systemtap with a pretty > old elfutils. Is is possible to try building systemtap with a later > version of elfutils? > I have version 0.13a installed on my system. Is that too old? I can give it a shot with latter versions. >> However, when i actually run uprobe-test, although both the probes >> fire successfully, the task generates a segmentation fault and dies on >> the 'test_func' probe. I see the following message on doing dmesg: > >> [ 1709.650827] Task died at uprobe probepoint: pid/tgid = 3934/3934, >> probepoint = 0x400517 > > > This means that the process died before the uprobe operation completed. > i.e either it was still running the handler or singlestep operation > hadnt occurred. > - Did you try using the core with gdb? Does address in bt and > disassemble output of test_func show the same address? Actually the uprobe is firing. I see the output of the following probe, when the function gets called: probe process("/home/gdhiman/tests/uprobe-test").function("test_func") { printf("test_func called!\n"); log($$parms); } However, the program immediately dies. The address of the function is 0x400517, the same as that shown in the probepoint. > > I am not able to see this problem. (I am running on a different kernel) > Which versions (systemtap and kernel) are you using? I might give them a shot as well. Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-15 9:11 ` Gaurav Dhiman @ 2010-06-15 17:46 ` Mark Wielaard 2010-06-16 13:54 ` Gaurav Dhiman 0 siblings, 1 reply; 19+ messages in thread From: Mark Wielaard @ 2010-06-15 17:46 UTC (permalink / raw) To: Gaurav Dhiman; +Cc: Srikar Dronamraju, systemtap On Mon, 2010-06-14 at 17:43 -0700, Gaurav Dhiman wrote: > > Looks to me that you are using a pretty recent systemtap with a pretty > > old elfutils. Is is possible to try building systemtap with a later > > version of elfutils? > > > I have version 0.13a installed on my system. Is that too old? I can > give it a shot with latter versions. That is pretty old. Some versions before 0.142 had bugs that could result in us misplacing probes. But that isn't the issue you are seeing here, since even if the probe was misplaced it should still not crash the binary you are inspecting. If possible, please do try using elfutils-0.142+ (latest is 0.147). Even though that shouldn't matter in your case. > > I am not able to see this problem. (I am running on a different kernel) > > > > Which versions (systemtap and kernel) are you using? I might give them > a shot as well. I was also unable to replicate. Using systemtap 1.2 or current git tip. elfutils 0.147. kernel 2.6.32-33.el6.x86_64 or 2.6.33.5-112.fc13.i686 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-15 17:46 ` Mark Wielaard @ 2010-06-16 13:54 ` Gaurav Dhiman 2010-06-16 19:05 ` Frank Ch. Eigler 0 siblings, 1 reply; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-16 13:54 UTC (permalink / raw) To: Mark Wielaard; +Cc: Srikar Dronamraju, systemtap On Tue, Jun 15, 2010 at 2:11 AM, Mark Wielaard <mjw@redhat.com> wrote: > On Mon, 2010-06-14 at 17:43 -0700, Gaurav Dhiman wrote: >> > Looks to me that you are using a pretty recent systemtap with a pretty >> > old elfutils. Is is possible to try building systemtap with a later >> > version of elfutils? >> > >> I have version 0.13a installed on my system. Is that too old? I can >> give it a shot with latter versions. > > That is pretty old. Some versions before 0.142 had bugs that could > result in us misplacing probes. But that isn't the issue you are seeing > here, since even if the probe was misplaced it should still not crash > the binary you are inspecting. > > If possible, please do try using elfutils-0.142+ (latest is 0.147). Even > though that shouldn't matter in your case. I have now upgraded to elfutils 0.143, and now the SIGSEGV is not happening anymore. But now it gives a Trace/breakpoint trap instead and the probe is removed just like it was happening before. The binary now likes like this: 0000000000400550 <test_func>: 400550: 53 push %rbx 400551: 89 fa mov %edi,%edx 400553: 89 fb mov %edi,%ebx 400555: be 7c 06 40 00 mov $0x40067c,%esi 40055a: bf 01 00 00 00 mov $0x1,%edi 40055f: 31 c0 xor %eax,%eax 400561: e8 da fe ff ff callq 400440 <__printf_chk@plt> 400566: 89 d8 mov %ebx,%eax 400568: 5b pop %rbx 400569: c3 retq 40056a: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1) And the systemtap debug messages that I see now are: stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[13414] addr 0000000000400550 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:3") stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[13414] reloc 0000000000400550 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:3") >> > I am not able to see this problem. (I am running on a different kernel) >> > >> >> Which versions (systemtap and kernel) are you using? I might give them >> a shot as well. > > I was also unable to replicate. Using systemtap 1.2 or current git tip. > elfutils 0.147. kernel 2.6.32-33.el6.x86_64 or 2.6.33.5-112.fc13.i686 > I am using kernel 2.6.29, which has the corresponding utrace patch. Can it be the reason that the older version is not compatible with the latest version of systemtap? Another thing that I forgot to mention in my previous emails is that I am running this kernel and systemtap inside a Xen DomU. Does that make a difference? Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-16 13:54 ` Gaurav Dhiman @ 2010-06-16 19:05 ` Frank Ch. Eigler 2010-06-22 8:57 ` Gaurav Dhiman 0 siblings, 1 reply; 19+ messages in thread From: Frank Ch. Eigler @ 2010-06-16 19:05 UTC (permalink / raw) To: Gaurav Dhiman; +Cc: Mark Wielaard, Srikar Dronamraju, systemtap Gaurav Dhiman <dimanuec@gmail.com> writes: > [...] > I am using kernel 2.6.29, which has the corresponding utrace patch. > Can it be the reason that the older version is not compatible with the > latest version of systemtap? It should work. I'm surprised that an elfutils version change produced anything visibly different. Perhaps you could try a fresh copy of 0.147 (and build systemtap with --with-elfutils=PATH)? > Another thing that I forgot to mention in my previous emails is that > I am running this kernel and systemtap inside a Xen DomU. Does that > make a difference? We have had some problems in the long-ago past with xen not emulating breakpoints inside the guests quite right. Would you be in a position to try kvm and/or hardware? - FChE ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-16 19:05 ` Frank Ch. Eigler @ 2010-06-22 8:57 ` Gaurav Dhiman 2010-06-22 10:36 ` Roland McGrath 0 siblings, 1 reply; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-22 8:57 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: Mark Wielaard, Srikar Dronamraju, systemtap On Wed, Jun 16, 2010 at 6:53 AM, Frank Ch. Eigler <fche@redhat.com> wrote: > Gaurav Dhiman <dimanuec@gmail.com> writes: > >> [...] >> I am using kernel 2.6.29, which has the corresponding utrace patch. >> Can it be the reason that the older version is not compatible with the >> latest version of systemtap? > > It should work. I'm surprised that an elfutils version change > produced anything visibly different. Perhaps you could try a fresh > copy of 0.147 (and build systemtap with --with-elfutils=PATH)? > >> Another thing that I forgot to mention in my previous emails is that >> I am running this kernel and systemtap inside a Xen DomU. Does that >> make a difference? > > We have had some problems in the long-ago past with xen not emulating > breakpoints inside the guests quite right. Would you be in a position > to try kvm and/or hardware? > Well, so I gave the whole thing a shot with KVM on the same machine. This time I ran with Linux kernel 2.6.34 (with utrace patch applied). I used systemtap with elfutils 0.147. And it worked (btw the 0.147 version did not work on the Xen DomU kernel)!! So I believe it might be either a Xen related issue or might have something to do with the fact that I am using Linux 2.6.34 now. I might verify the latter by running 2.6.29 inside KVM. For reasons specific to our setup, we cannot go beyond 2.6.29 for the Xen DomU. I have also observed, that around 3 hunks of the utrace patch always fail in tracehook.h. I typically apply them manually in the functions they are supposed to go into. Is the patch not actively updated, or these changes are outdated? I have appended my tracehook.h.rej below. *************** static inline __must_check int tracehook *** 134,139 **** */ static inline void tracehook_report_syscall_exit(struct pt_regs *regs, int step) { if (step && (task_ptrace(current) & PT_PTRACED)) { siginfo_t info; user_single_step_siginfo(current, regs, &info); --- 140,148 ---- */ static inline void tracehook_report_syscall_exit(struct pt_regs *regs, int step) { + if (task_utrace_flags(current) & UTRACE_EVENT(SYSCALL_EXIT)) + utrace_report_syscall_exit(regs); + if (step && (task_ptrace(current) & PT_PTRACED)) { siginfo_t info; user_single_step_siginfo(current, regs, &info); *************** static inline void tracehook_signal_hand *** 386,391 **** const struct k_sigaction *ka, struct pt_regs *regs, int stepping) { if (stepping && (task_ptrace(current) & PT_PTRACED)) ptrace_notify(SIGTRAP); } --- 434,441 ---- const struct k_sigaction *ka, struct pt_regs *regs, int stepping) { + if (task_utrace_flags(current)) + utrace_signal_handler(current, stepping); if (stepping && (task_ptrace(current) & PT_PTRACED)) ptrace_notify(SIGTRAP); } *************** static inline int tracehook_get_signal(s *** 492,497 **** */ static inline int tracehook_notify_jctl(int notify, int why) { return notify ?: task_ptrace(current) ? why : 0; } --- 551,558 ---- */ static inline int tracehook_notify_jctl(int notify, int why) { + if (task_utrace_flags(current) & UTRACE_EVENT(JCTL)) + utrace_report_jctl(notify, why); return notify ?: task_ptrace(current) ? why : 0; } Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-22 8:57 ` Gaurav Dhiman @ 2010-06-22 10:36 ` Roland McGrath 2010-06-22 13:05 ` Gaurav Dhiman 0 siblings, 1 reply; 19+ messages in thread From: Roland McGrath @ 2010-06-22 10:36 UTC (permalink / raw) To: Gaurav Dhiman Cc: Frank Ch. Eigler, Mark Wielaard, Srikar Dronamraju, systemtap > I have also observed, that around 3 hunks of the utrace patch always > fail in tracehook.h. I typically apply them manually in the functions > they are supposed to go into. Is the patch not actively updated, or > these changes are outdated? I have appended my tracehook.h.rej below. What baseline kernel and what utrace patch are you using? At http://people.redhat.com/~roland/utrace/ and git://git.kernel.org/pub/scm/linux/kernel/git/frob/linux-2.6-utrace.git there are current patch-series/branches for 2.6-current (v2.6.35-rc3), 2.6.34 (e.g. rawhide), 2.6.33 (e.g. F-13), and 2.6.32 (e.g. F-12). I just did the merge to update the 2.6-current to 2.6.35-rc3 from somewhere in the post-2.6.34/pre-rc1 period, but it was automatic with no conflicts. Thanks, Roland ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-22 10:36 ` Roland McGrath @ 2010-06-22 13:05 ` Gaurav Dhiman 2010-06-22 13:11 ` Roland McGrath 0 siblings, 1 reply; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-22 13:05 UTC (permalink / raw) To: Roland McGrath Cc: Frank Ch. Eigler, Mark Wielaard, Srikar Dronamraju, systemtap On Mon, Jun 21, 2010 at 6:43 PM, Roland McGrath <roland@redhat.com> wrote: >> I have also observed, that around 3 hunks of the utrace patch always >> fail in tracehook.h. I typically apply them manually in the functions >> they are supposed to go into. Is the patch not actively updated, or >> these changes are outdated? I have appended my tracehook.h.rej below. > > What baseline kernel and what utrace patch are you using? > > At http://people.redhat.com/~roland/utrace/ > and git://git.kernel.org/pub/scm/linux/kernel/git/frob/linux-2.6-utrace.git > there are current patch-series/branches for 2.6-current (v2.6.35-rc3), > 2.6.34 (e.g. rawhide), 2.6.33 (e.g. F-13), and 2.6.32 (e.g. F-12). > > I just did the merge to update the 2.6-current to 2.6.35-rc3 from somewhere > in the post-2.6.34/pre-rc1 period, but it was automatic with no conflicts. > I got the 2.6.34 kernel from kernel.org, and the following utrace patch: http://people.redhat.com/roland/utrace/2.6.34/utrace.patch Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-22 13:05 ` Gaurav Dhiman @ 2010-06-22 13:11 ` Roland McGrath 2010-06-22 13:12 ` Gaurav Dhiman 0 siblings, 1 reply; 19+ messages in thread From: Roland McGrath @ 2010-06-22 13:11 UTC (permalink / raw) To: Gaurav Dhiman Cc: Frank Ch. Eigler, Mark Wielaard, Srikar Dronamraju, systemtap > I got the 2.6.34 kernel from kernel.org, and the following utrace patch: > http://people.redhat.com/roland/utrace/2.6.34/utrace.patch Follow the series file in that directory to apply the patches in proper order. Thanks, Roland ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-22 13:11 ` Roland McGrath @ 2010-06-22 13:12 ` Gaurav Dhiman 0 siblings, 0 replies; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-22 13:12 UTC (permalink / raw) To: Roland McGrath Cc: Frank Ch. Eigler, Mark Wielaard, Srikar Dronamraju, systemtap On Mon, Jun 21, 2010 at 9:07 PM, Roland McGrath <roland@redhat.com> wrote: >> I got the 2.6.34 kernel from kernel.org, and the following utrace patch: >> http://people.redhat.com/roland/utrace/2.6.34/utrace.patch > > Follow the series file in that directory to apply the patches in proper order. > My bad, I did not get those failed hunk messages when I followed the order in the series file! Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint [not found] ` <AANLkTilvvj9hdiaa9VWjO0j-jK02LMweg5cAEBgoLyRK@mail.gmail.com> 2010-06-15 9:11 ` Gaurav Dhiman @ 2010-06-15 18:17 ` Srikar Dronamraju 2010-06-16 14:19 ` Gaurav Dhiman 1 sibling, 1 reply; 19+ messages in thread From: Srikar Dronamraju @ 2010-06-15 18:17 UTC (permalink / raw) To: Gaurav Dhiman; +Cc: SystemTap ML > > Which versions (systemtap and kernel) are you using? I might give them > a shot as well. I run 2.6.32 kernels with latest snapshot of systemtap and elfutils. Can you try removing the log($$parms) statement and see if it still fails? I dont see a reason why it should make a difference but its just a wild wild guess. -- Thanks and Regards Srikar ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-15 18:17 ` Srikar Dronamraju @ 2010-06-16 14:19 ` Gaurav Dhiman 0 siblings, 0 replies; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-16 14:19 UTC (permalink / raw) To: Srikar Dronamraju; +Cc: SystemTap ML On Tue, Jun 15, 2010 at 3:37 AM, Srikar Dronamraju <srikar@linux.vnet.ibm.com> wrote: >> >> Which versions (systemtap and kernel) are you using? I might give them >> a shot as well. > > I run 2.6.32 kernels with latest snapshot of systemtap and elfutils. > > Can you try removing the log($$parms) statement and see if it still > fails? I dont see a reason why it should make a difference but its just > a wild wild guess. No, it did not make any difference! Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-14 9:18 Segmentation Fault: Task died at uprobe probepoint Gaurav Dhiman 2010-06-14 12:55 ` Srikar Dronamraju @ 2010-06-14 21:06 ` Frank Ch. Eigler 2010-06-15 9:24 ` Gaurav Dhiman 1 sibling, 1 reply; 19+ messages in thread From: Frank Ch. Eigler @ 2010-06-14 21:06 UTC (permalink / raw) To: Gaurav Dhiman; +Cc: systemtap Gaurav Dhiman <dimanuec@gmail.com> writes: > [...] > I have recently installed SystemTap on my Ubuntu 9.04 based system. I > have Linux kernel 2.6.29.1 with utrace support [...] Great. > [...] > However, when i actually run uprobe-test, although both the probes > fire successfully, the task generates a segmentation fault and dies on > the 'test_func' probe. I see the following message on doing dmesg: > [ 1709.650827] Task died at uprobe probepoint: pid/tgid = 3934/3934, > probepoint = 0x400517 Almost certainly it's a uprobes or utrace or systemtap bug, rather than something you might have done wrong. Can you collect more information, such as systemtap git version, stap -vvvt -DDEBUG_UPROBES traces, disassembly of your probe-target executable? - FChE ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-14 21:06 ` Frank Ch. Eigler @ 2010-06-15 9:24 ` Gaurav Dhiman 2010-06-15 10:39 ` Frank Ch. Eigler 0 siblings, 1 reply; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-15 9:24 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: systemtap On Mon, Jun 14, 2010 at 9:56 AM, Frank Ch. Eigler <fche@redhat.com> wrote: > Gaurav Dhiman <dimanuec@gmail.com> writes: > >> [...] >> I have recently installed SystemTap on my Ubuntu 9.04 based system. I >> have Linux kernel 2.6.29.1 with utrace support [...] > > Great. > >> [...] >> However, when i actually run uprobe-test, although both the probes >> fire successfully, the task generates a segmentation fault and dies on >> the 'test_func' probe. I see the following message on doing dmesg: >> [ 1709.650827] Task died at uprobe probepoint: pid/tgid = 3934/3934, >> probepoint = 0x400517 > > Almost certainly it's a uprobes or utrace or systemtap bug, rather > than something you might have done wrong. Can you collect more > information, such as systemtap git version, stap -vvvt -DDEBUG_UPROBES > traces, disassembly of your probe-target executable? The following commit is on the head of the git log I have (I don't know how to get the exact systemtap version): commit b275171e26c0cf9441858209ac62e70b8ddda71d Author: Josh Stone <jistone@redhat.com> Date: Thu Jun 10 16:05:29 2010 -0700 PR11690: Remove a dtrace debug-print, and fix probe_perf Stan's commit 278c975 fixes the core issue; I'm just polishing... * dtrace.in: Comment out the debug-print before calling gcc. * scripts/probe_perf/bench.sh: Pass -DSTAP_SDT_V2 to dtrace. Following are the logs from the Pass 5 of the run with stap -vvvt -DDEBUG_UPROBES: Pass 5: starting run. Running /usr/local/bin/staprun -v -v -u /tmp/stapmfXZax/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686.ko staprun:main:273 modpath="/tmp/stapmfXZax/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686.ko", modname="stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686" staprun:init_staprun:211 init_staprun spawning: /bin/grep -q unregister_uprobe /proc/kallsyms staprun:insert_module:60 inserting module staprun:insert_module:79 module options: _stp_bufsize=0 staprun:init_ctl_channel:31 Opened /sys/kernel/debug/systemtap/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686/.cmd (3) staprun:close_ctl_channel:50 Closed ctl fd 3 execing: /usr/local/libexec/systemtap/stapio -v -v -u /tmp/stapmfXZax/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686.ko stapio:main:37 modpath="/tmp/stapmfXZax/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686.ko", modname="stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686" stapio:init_stapio:317 init_stapio stapio:init_ctl_channel:31 Opened /sys/kernel/debug/systemtap/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686/.cmd (3) stapio:stp_main_loop:494 in main loop stapio:init_relayfs:238 initializing relayfs stapio:init_relayfs:262 attempting to open /sys/kernel/debug/systemtap/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686/trace0 stapio:init_relayfs:262 attempting to open /sys/kernel/debug/systemtap/stap_0ebd34b5ef2ddf9f38803a661fa38a13_1686/trace1 stapio:init_relayfs:268 ncpus=1, bulkmode = 0 stapio:init_relayfs:345 starting threads stapio:stp_main_loop:571 probe_start() returned 0 stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[25763] addr 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") test begin!! test_func called! i=0x5 stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[25763] reloc 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") The program receives a SIGSEGV signal at this point. By the probe target executable do you mean, the .ko file that is generated? The disassembled file of that is pretty big, close to 17000 lines. Do you want that or something else? Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-15 9:24 ` Gaurav Dhiman @ 2010-06-15 10:39 ` Frank Ch. Eigler 2010-06-15 12:25 ` Gaurav Dhiman 0 siblings, 1 reply; 19+ messages in thread From: Frank Ch. Eigler @ 2010-06-15 10:39 UTC (permalink / raw) To: Gaurav Dhiman; +Cc: systemtap Gaurav Dhiman <dimanuec@gmail.com> writes: > [...] > The following commit is on the head of the git log I have (I don't > know how to get the exact systemtap version): > > commit b275171e26c0cf9441858209ac62e70b8ddda71d > [...] That's perfectly specific, thanks. > [...] > Following are the logs from the Pass 5 of the run with stap -vvvt > -DDEBUG_UPROBES: > > Pass 5: starting run. > [...] > stapio:stp_main_loop:571 probe_start() returned 0 > stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[25763] addr 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") > test begin!! > test_func called! > i=0x5 > stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[25763] reloc 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") > > The program receives a SIGSEGV signal at this point. OK, something is wrong here. See that -uprobe line? It means stap has for some reason determined that it's time to remove the uprobe, but that should not happen until either the script or the target program exits. This will make things more verbose, but could you try adding -DDEBUG_TASK_FINDER and -DDEBUG_TASK_FINDER_VMA to the stap command line? > By the probe target executable do you mean, the .ko file that is > generated? The disassembled file of that is pretty big, close to 17000 > lines. Do you want that or something else? I meant the uprobe-test binary, just to see what instructions were in the vicinity of the test_func function. Just that area. - FChE ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-15 10:39 ` Frank Ch. Eigler @ 2010-06-15 12:25 ` Gaurav Dhiman 2010-06-15 18:07 ` Mark Wielaard 0 siblings, 1 reply; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-15 12:25 UTC (permalink / raw) To: Frank Ch. Eigler; +Cc: systemtap On Mon, Jun 14, 2010 at 6:33 PM, Frank Ch. Eigler <fche@redhat.com> wrote: > Gaurav Dhiman <dimanuec@gmail.com> writes: > >> [...] >> The following commit is on the head of the git log I have (I don't >> know how to get the exact systemtap version): >> >> commit b275171e26c0cf9441858209ac62e70b8ddda71d >> [...] > > That's perfectly specific, thanks. > >> [...] >> Following are the logs from the Pass 5 of the run with stap -vvvt >> -DDEBUG_UPROBES: >> >> Pass 5: starting run. >> [...] >> stapio:stp_main_loop:571 probe_start() returned 0 >> stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[25763] addr 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") >> test begin!! >> test_func called! >> i=0x5 >> stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[25763] reloc 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") >> >> The program receives a SIGSEGV signal at this point. > > OK, something is wrong here. See that -uprobe line? It means stap > has for some reason determined that it's time to remove the uprobe, > but that should not happen until either the script or the target > program exits. > > This will make things more verbose, but could you try adding > -DDEBUG_TASK_FINDER and -DDEBUG_TASK_FINDER_VMA to the stap > command line? > You are right, that looks suspicious; it is actually removing the probe, when it should not be doing it. Here is the more verbose output: stap_uprobe_process_found:246: +proc pid 27237 stf ffffffffa001dce0 ffffffffa001dce0 path /home/gdhiman/tests/uprobe-test stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[27237] addr 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") test begin!! _stp_tf_exec_cb:29: tsk 27237:27237 , register_p: 1, process_p: 1 __stp_call_mmap_callbacks:599: pid 27237, a/l/o/p/path 0x400000 0x1000 0x0 r-xp /home/gdhiman/tests/uprobe-test _stp_tf_mmap_cb:51: mmap_cb: tsk 27237:27237 path /home/gdhiman/tests/uprobe-test, addr 0x00400000, length 0x00001000, offset 0x0, flags 0x8001875 _stp_tf_mmap_cb:63: vm_cb: matched path /home/gdhiman/tests/uprobe-test to module (for sec: .absolute) __stp_call_mmap_callbacks:599: pid 27237, a/l/o/p/path 0x600000 0x2000 0x0 rw-p /home/gdhiman/tests/uprobe-test _stp_tf_mmap_cb:51: mmap_cb: tsk 27237:27237 path /home/gdhiman/tests/uprobe-test, addr 0x00600000, length 0x00002000, offset 0x0, flags 0x8101873 __stp_call_mmap_callbacks:599: pid 27237, a/l/o/p/path 0x7faadb80e000 0x20000 0x0 r-xp /lib/ld-2.9.so _stp_tf_mmap_cb:51: mmap_cb: tsk 27237:27237 path /lib/ld-2.9.so, addr 0x7faadb80e000, length 0x00020000, offset 0x0, flags 0x8000875 _stp_tf_mmap_cb:92: registered 'ld-2.9.so' for 27237 (res:0) __stp_call_mmap_callbacks:599: pid 27237, a/l/o/p/path 0x7faadba2d000 0x2000 0x1f000 rw-p /lib/ld-2.9.so _stp_tf_mmap_cb:51: mmap_cb: tsk 27237:27237 path /lib/ld-2.9.so, addr 0x7faadba2d000, length 0x00002000, offset 0x1f000, flags 0x8100873 __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found mmap(0x0), returned 0x7faadba2c000 __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found mmap(0x0), returned 0x7faadba2a000 __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found mmap(0x0), returned 0x7faadba25000 __stp_call_mmap_callbacks:599: pid 27237, a/l/o/p/path 0x7faadba25000 0x5000 0x0 r--p /etc/ld.so.cache _stp_tf_mmap_cb:51: mmap_cb: tsk 27237:27237 path /etc/ld.so.cache, addr 0x7faadba25000, length 0x00005000, offset 0x0, flags 0x8000071 __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found mmap(0x0), returned 0x7faadb49c000 __stp_call_mmap_callbacks:599: pid 27237, a/l/o/p/path 0x7faadb49c000 0x372000 0x0 r-xp /lib/libc-2.9.so _stp_tf_mmap_cb:51: mmap_cb: tsk 27237:27237 path /lib/libc-2.9.so, addr 0x7faadb49c000, length 0x00372000, offset 0x0, flags 0x8000075 _stp_tf_mmap_cb:92: registered 'libc-2.9.so' for 27237 (res:0) __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found mmap(0x0), returned 0x7faadb804000 __stp_call_mmap_callbacks:599: pid 27237, a/l/o/p/path 0x7faadb804000 0x5000 0x168000 rw-p /lib/libc-2.9.so _stp_tf_mmap_cb:51: mmap_cb: tsk 27237:27237 path /lib/libc-2.9.so, addr 0x7faadb804000, length 0x00005000, offset 0x168000, flags 0x8100073 __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found mmap(0x0), returned 0x7faadb809000 __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found mmap(0x0), returned 0x7faadba24000 __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found mmap(0x0), returned 0x7faadba23000 __stp_utrace_task_finder_target_syscall_exit:1423: tsk 27237 found munmap(0x7faadba25000), returned 0x0 test_func called! i=0x5 stap_uprobe_process_found:246: -proc pid 27237 stf ffffffffa001dce0 ffffffffa001dce0 path /home/gdhiman/tests/uprobe-test stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[27237] reloc 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") _stp_tf_exec_cb:29: tsk 27237:27237 , register_p: 0, process_p: 1 > > I meant the uprobe-test binary, just to see what instructions were > in the vicinity of the test_func function. Just that area. > Here it is: 000000000040050c <test_func>: 40050c: 55 push %rbp 40050d: 48 89 e5 mov %rsp,%rbp 400510: 48 83 ec 10 sub $0x10,%rsp 400514: 89 7d fc mov %edi,-0x4(%rbp) 400517: 8b 75 fc mov -0x4(%rbp),%esi 40051a: bf 3c 06 40 00 mov $0x40063c,%edi 40051f: b8 00 00 00 00 mov $0x0,%eax 400524: e8 cf fe ff ff callq 4003f8 <printf@plt> 400529: 8b 45 fc mov -0x4(%rbp),%eax 40052c: c9 leaveq 40052d: c3 retq 000000000040052e <main>: 40052e: 55 push %rbp 40052f: 48 89 e5 mov %rsp,%rbp 400532: 48 83 ec 10 sub $0x10,%rsp 400536: bf 05 00 00 00 mov $0x5,%edi 40053b: e8 cc ff ff ff callq 40050c <test_func> 400540: 89 45 fc mov %eax,-0x4(%rbp) 400543: b8 00 00 00 00 mov $0x0,%eax 400548: c9 leaveq 400549: c3 retq 40054a: 90 nop 40054b: 90 nop 40054c: 90 nop 40054d: 90 nop 40054e: 90 nop 40054f: 90 nop Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-15 12:25 ` Gaurav Dhiman @ 2010-06-15 18:07 ` Mark Wielaard 2010-06-16 14:08 ` Gaurav Dhiman 0 siblings, 1 reply; 19+ messages in thread From: Mark Wielaard @ 2010-06-15 18:07 UTC (permalink / raw) To: Gaurav Dhiman; +Cc: Frank Ch. Eigler, systemtap On Mon, 2010-06-14 at 20:52 -0700, Gaurav Dhiman wrote: > On Mon, Jun 14, 2010 at 6:33 PM, Frank Ch. Eigler <fche@redhat.com> wrote: > > Gaurav Dhiman <dimanuec@gmail.com> writes: > >> [...] > >> Following are the logs from the Pass 5 of the run with stap -vvvt > >> -DDEBUG_UPROBES: > >> > >> Pass 5: starting run. > >> [...] > >> stapio:stp_main_loop:571 probe_start() returned 0 > >> stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[25763] addr 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") > >> test begin!! > >> test_func called! > >> i=0x5 > >> stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[25763] reloc 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") > >> > >> The program receives a SIGSEGV signal at this point. > > > > OK, something is wrong here. See that -uprobe line? It means stap > > has for some reason determined that it's time to remove the uprobe, > > but that should not happen until either the script or the target > > program exits. > > > > This will make things more verbose, but could you try adding > > -DDEBUG_TASK_FINDER and -DDEBUG_TASK_FINDER_VMA to the stap > > command line? > > > You are right, that looks suspicious; it is actually removing the > probe, when it should not be doing it. Here is the more verbose > output: > > test_func called! > i=0x5 > stap_uprobe_process_found:246: -proc pid 27237 stf ffffffffa001dce0 > ffffffffa001dce0 path /home/gdhiman/tests/uprobe-test > stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process > uprobe-test[27237] reloc 0000000000400517 pp > process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") > _stp_tf_exec_cb:29: tsk 27237:27237 , register_p: 0, process_p: 1 Unless I am reading the logs wrong, I do think this is expected. The process gets a SIGSEGV and so uprobes is notified that it needs to clean up (-proc pid 27237 means process is gone). > > I meant the uprobe-test binary, just to see what instructions were > > in the vicinity of the test_func function. Just that area. > > > > Here it is: > > 000000000040050c <test_func>: > 40050c: 55 push %rbp > 40050d: 48 89 e5 mov %rsp,%rbp > 400510: 48 83 ec 10 sub $0x10,%rsp > 400514: 89 7d fc mov %edi,-0x4(%rbp) > 400517: 8b 75 fc mov -0x4(%rbp),%esi > 40051a: bf 3c 06 40 00 mov $0x40063c,%edi > 40051f: b8 00 00 00 00 mov $0x0,%eax > 400524: e8 cf fe ff ff callq 4003f8 <printf@plt> > 400529: 8b 45 fc mov -0x4(%rbp),%eax > 40052c: c9 leaveq > 40052d: c3 retq That does look OK, even though it looks different from my local build: Dump of assembler code for function test_func: 0x080483d0 <+0>: push %ebp 0x080483d1 <+1>: mov %esp,%ebp 0x080483d3 <+3>: push %ebx 0x080483d4 <+4>: sub $0x14,%esp 0x080483d7 <+7>: mov 0x8(%ebp),%ebx 0x080483da <+10>: movl $0x80484f4,(%esp) 0x080483e1 <+17>: mov %ebx,0x4(%esp) 0x080483e5 <+21>: call 0x80482f4 <printf@plt> 0x080483ea <+26>: mov %ebx,%eax 0x080483ec <+28>: add $0x14,%esp 0x080483ef <+31>: pop %ebx 0x080483f0 <+32>: pop %ebp 0x080483f1 <+33>: ret End of assembler dump. What compiler and flags did you use? Here gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) with -g -O2. If possible could you set ulimit -c unlimited and catch the core file of the program and give the disassembly from test_func in the core? Maybe the breakpoint instruction in there was misplaced? Thanks, Mark ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Segmentation Fault: Task died at uprobe probepoint 2010-06-15 18:07 ` Mark Wielaard @ 2010-06-16 14:08 ` Gaurav Dhiman 0 siblings, 0 replies; 19+ messages in thread From: Gaurav Dhiman @ 2010-06-16 14:08 UTC (permalink / raw) To: Mark Wielaard; +Cc: Frank Ch. Eigler, systemtap On Tue, Jun 15, 2010 at 2:23 AM, Mark Wielaard <mjw@redhat.com> wrote: > On Mon, 2010-06-14 at 20:52 -0700, Gaurav Dhiman wrote: >> On Mon, Jun 14, 2010 at 6:33 PM, Frank Ch. Eigler <fche@redhat.com> wrote: >> > Gaurav Dhiman <dimanuec@gmail.com> writes: >> >> [...] >> >> Following are the logs from the Pass 5 of the run with stap -vvvt >> >> -DDEBUG_UPROBES: >> >> >> >> Pass 5: starting run. >> >> [...] >> >> stapio:stp_main_loop:571 probe_start() returned 0 >> >> stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[25763] addr 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") >> >> test begin!! >> >> test_func called! >> >> i=0x5 >> >> stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[25763] reloc 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") >> >> >> >> The program receives a SIGSEGV signal at this point. >> > >> > OK, something is wrong here. See that -uprobe line? It means stap >> > has for some reason determined that it's time to remove the uprobe, >> > but that should not happen until either the script or the target >> > program exits. >> > >> > This will make things more verbose, but could you try adding >> > -DDEBUG_TASK_FINDER and -DDEBUG_TASK_FINDER_VMA to the stap >> > command line? >> > >> You are right, that looks suspicious; it is actually removing the >> probe, when it should not be doing it. Here is the more verbose >> output: >> >> test_func called! >> i=0x5 >> stap_uprobe_process_found:246: -proc pid 27237 stf ffffffffa001dce0 >> ffffffffa001dce0 path /home/gdhiman/tests/uprobe-test >> stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process >> uprobe-test[27237] reloc 0000000000400517 pp >> process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4") >> _stp_tf_exec_cb:29: tsk 27237:27237 , register_p: 0, process_p: 1 > > Unless I am reading the logs wrong, I do think this is expected. The > process gets a SIGSEGV and so uprobes is notified that it needs to clean > up (-proc pid 27237 means process is gone). > >> > I meant the uprobe-test binary, just to see what instructions were >> > in the vicinity of the test_func function. Just that area. >> > >> >> Here it is: >> >> 000000000040050c <test_func>: >> 40050c: 55 push %rbp >> 40050d: 48 89 e5 mov %rsp,%rbp >> 400510: 48 83 ec 10 sub $0x10,%rsp >> 400514: 89 7d fc mov %edi,-0x4(%rbp) >> 400517: 8b 75 fc mov -0x4(%rbp),%esi >> 40051a: bf 3c 06 40 00 mov $0x40063c,%edi >> 40051f: b8 00 00 00 00 mov $0x0,%eax >> 400524: e8 cf fe ff ff callq 4003f8 <printf@plt> >> 400529: 8b 45 fc mov -0x4(%rbp),%eax >> 40052c: c9 leaveq >> 40052d: c3 retq > > That does look OK, even though it looks different from my local build: > > Dump of assembler code for function test_func: > 0x080483d0 <+0>: push %ebp > 0x080483d1 <+1>: mov %esp,%ebp > 0x080483d3 <+3>: push %ebx > 0x080483d4 <+4>: sub $0x14,%esp > 0x080483d7 <+7>: mov 0x8(%ebp),%ebx > 0x080483da <+10>: movl $0x80484f4,(%esp) > 0x080483e1 <+17>: mov %ebx,0x4(%esp) > 0x080483e5 <+21>: call 0x80482f4 <printf@plt> > 0x080483ea <+26>: mov %ebx,%eax > 0x080483ec <+28>: add $0x14,%esp > 0x080483ef <+31>: pop %ebx > 0x080483f0 <+32>: pop %ebp > 0x080483f1 <+33>: ret > End of assembler dump. > > What compiler and flags did you use? Here gcc version 4.4.4 20100503 > (Red Hat 4.4.4-2) with -g -O2. > I am using gcc 4.4.3 and using -g -O2 as well. > If possible could you set ulimit -c unlimited and catch the core file of > the program and give the disassembly from test_func in the core? Maybe > the breakpoint instruction in there was misplaced? I will try to send this tomorrow. Thanks, -Gaurav ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2010-06-22 7:01 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-06-14 9:18 Segmentation Fault: Task died at uprobe probepoint Gaurav Dhiman 2010-06-14 12:55 ` Srikar Dronamraju [not found] ` <AANLkTilvvj9hdiaa9VWjO0j-jK02LMweg5cAEBgoLyRK@mail.gmail.com> 2010-06-15 9:11 ` Gaurav Dhiman 2010-06-15 17:46 ` Mark Wielaard 2010-06-16 13:54 ` Gaurav Dhiman 2010-06-16 19:05 ` Frank Ch. Eigler 2010-06-22 8:57 ` Gaurav Dhiman 2010-06-22 10:36 ` Roland McGrath 2010-06-22 13:05 ` Gaurav Dhiman 2010-06-22 13:11 ` Roland McGrath 2010-06-22 13:12 ` Gaurav Dhiman 2010-06-15 18:17 ` Srikar Dronamraju 2010-06-16 14:19 ` Gaurav Dhiman 2010-06-14 21:06 ` Frank Ch. Eigler 2010-06-15 9:24 ` Gaurav Dhiman 2010-06-15 10:39 ` Frank Ch. Eigler 2010-06-15 12:25 ` Gaurav Dhiman 2010-06-15 18:07 ` Mark Wielaard 2010-06-16 14:08 ` Gaurav Dhiman
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).