* [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault
@ 2019-08-05 16:43 avi@cloudius-systems.com
2019-08-23 22:21 ` [Bug dyninst/24884] " fche at redhat dot com
` (9 more replies)
0 siblings, 10 replies; 11+ messages in thread
From: avi@cloudius-systems.com @ 2019-08-05 16:43 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
Bug ID: 24884
Summary: stapdyn crashes with a segmentation fault
Product: systemtap
Version: unspecified
Status: UNCONFIRMED
Severity: normal
Priority: P2
Component: dyninst
Assignee: systemtap at sourceware dot org
Reporter: avi@cloudius-systems.com
Target Milestone: ---
Trying the following script:
#!/usr/bin/stap
# usage: task-histogram.stap process_name
global hist
probe process.mark("reactor_run_tasks_single_start") {
++hist[tid(), $arg1]
}
probe end {
foreach ([tid, addr] in hist) {
printf("%10d %8d 0x%x\n", hist[tid, addr], tid, addr)
}
}
(trying to collect a histogram of tasks)
With this command line:
stap --dyninst -x $(pgrep -x httpd) ./debug/task-histogram.stap
Crashes with
WARNING: /usr/bin/stapdyn exited with signal: 11 (Segmentation fault)
systemtap-4.1-1.fc30.x86_64
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
@ 2019-08-23 22:21 ` fche at redhat dot com
2019-11-12 13:08 ` avi@cloudius-systems.com
` (8 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: fche at redhat dot com @ 2019-08-23 22:21 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |fche at redhat dot com
--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
Hi, Avi, sorry for not noticing this earlier. Some questions to assist in the
local reproduction of this problem:
- arch: x86-64?
- same script works in lkm (non-dyninst) mode?
- tried stap -p4 --dyninst FOO.stp ; gdb -args stapbpf FOO.so
so as to get a gdb backtrace at the crash site?
- what level of traffic is the httpd process absorbing during this time?
(thus: how much thread / child-process changes?)
- tried targeting a program other than this httpd?
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
2019-08-23 22:21 ` [Bug dyninst/24884] " fche at redhat dot com
@ 2019-11-12 13:08 ` avi@cloudius-systems.com
2019-11-12 13:11 ` avi@cloudius-systems.com
` (7 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: avi@cloudius-systems.com @ 2019-11-12 13:08 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
--- Comment #2 from Avi Kivity <avi@cloudius-systems.com> ---
Sorry for noticing _your_ comment so late. I retested with
systemtap-4.1-2.fc30.x86_64, and it appears to work.
(x86_64, don't remember if I tried lkm, httpd has no forks/pthread_creates at
all)
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
2019-08-23 22:21 ` [Bug dyninst/24884] " fche at redhat dot com
2019-11-12 13:08 ` avi@cloudius-systems.com
@ 2019-11-12 13:11 ` avi@cloudius-systems.com
2019-11-12 13:21 ` avi@cloudius-systems.com
` (6 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: avi@cloudius-systems.com @ 2019-11-12 13:11 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
--- Comment #3 from Avi Kivity <avi@cloudius-systems.com> ---
The performance impact is horrendous however, 5X slower (251k req/sec without
the script, 47k with the script). Does dyninst rewrite the entire program or
just the entry points to the probe?
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
` (2 preceding siblings ...)
2019-11-12 13:11 ` avi@cloudius-systems.com
@ 2019-11-12 13:21 ` avi@cloudius-systems.com
2019-11-12 13:25 ` avi@cloudius-systems.com
` (5 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: avi@cloudius-systems.com @ 2019-11-12 13:21 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
--- Comment #4 from Avi Kivity <avi@cloudius-systems.com> ---
And now I get segmentation faults again.
#0 int_process::removeAllBreakpoints (this=0x55bf9c39eab8) at
/usr/include/c++/9/bits/stl_tree.h:208
#1 0x00007f0c1737589f in linux_process::preTerminate (this=0x55bf9c39e820) at
/usr/src/debug/dyninst-10.0.0-7.fc30.x86_64/dyninst-10.0.0/proccontrol/src/linux.C:1740
#2 0x00007f0c1733a4d1 in Dyninst::ProcControlAPI::ProcessSet::terminate
(this=0x55bfb12f09e0) at
/usr/src/debug/dyninst-10.0.0-7.fc30.x86_64/dyninst-10.0.0/proccontrol/src/procset.C:1644
#3 0x00007f0c172d95db in Dyninst::ProcControlAPI::Process::terminate
(this=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:732
#4 0x00007f0c17da7ab5 in PCProcess::terminateProcess (this=0x55bfa0f37cb0) at
/usr/include/boost/smart_ptr/shared_ptr.hpp:732
#5 PCProcess::terminateProcess (this=0x55bfa0f37cb0) at
/usr/src/debug/dyninst-10.0.0-7.fc30.x86_64/dyninst-10.0.0/dyninstAPI/src/dynProcess.C:1027
#6 0x00007f0c17db377c in PCProcess::attachProcess (progpath=..., pid=16581,
analysisMode=BPatch_normalMode) at
/usr/src/debug/dyninst-10.0.0-7.fc30.x86_64/dyninst-10.0.0/dyninstAPI/src/dynProcess.C:162
#7 0x00007f0c17cf269a in BPatch_process::BPatch_process(char const*, int,
BPatch_hybridMode) () at
/usr/src/debug/dyninst-10.0.0-7.fc30.x86_64/dyninst-10.0.0/dyninstAPI/src/BPatch_process.C:328
#8 0x00007f0c17ccf027 in BPatch::processAttach (this=<optimized out>,
path=0x0, pid=16581, mode=BPatch_normalMode) at
/usr/src/debug/dyninst-10.0.0-7.fc30.x86_64/dyninst-10.0.0/dyninstAPI/src/BPatch.C:1260
#9 0x000055bf9abb4519 in ?? ()
#10 0x000055bf9c362440 in ?? ()
#11 0x00007ffd51a63ec0 in ?? ()
#12 0x00000000000040c5 in probe_13397 ()
#13 0x0000000000000000 in ?? ()
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
` (3 preceding siblings ...)
2019-11-12 13:21 ` avi@cloudius-systems.com
@ 2019-11-12 13:25 ` avi@cloudius-systems.com
2019-11-12 13:38 ` avi@cloudius-systems.com
` (4 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: avi@cloudius-systems.com @ 2019-11-12 13:25 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
--- Comment #5 from Avi Kivity <avi@cloudius-systems.com> ---
I think the trigger for the crash is re-attaching to a process after detaching
from it.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
` (4 preceding siblings ...)
2019-11-12 13:25 ` avi@cloudius-systems.com
@ 2019-11-12 13:38 ` avi@cloudius-systems.com
2019-11-23 1:26 ` fche at redhat dot com
` (3 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: avi@cloudius-systems.com @ 2019-11-12 13:38 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
--- Comment #6 from Avi Kivity <avi@cloudius-systems.com> ---
And the cause for the slowness is lock contention. With only two threads.
Please please please add thread-local storage to the language.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
` (5 preceding siblings ...)
2019-11-12 13:38 ` avi@cloudius-systems.com
@ 2019-11-23 1:26 ` fche at redhat dot com
2019-11-24 8:57 ` avi@cloudius-systems.com
` (2 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: fche at redhat dot com @ 2019-11-23 1:26 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
--- Comment #7 from Frank Ch. Eigler <fche at redhat dot com> ---
(In reply to Avi Kivity from comment #6)
> And the cause for the slowness is lock contention. With only two threads.
> Please please please add thread-local storage to the language.
It's more of a runtime issue than a language issue, but yeah. Surely there are
some optimization opportunities in what we emit for:
probe process.mark("reactor_run_tasks_single_start") {
++hist[tid(), $arg1]
}
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
` (6 preceding siblings ...)
2019-11-23 1:26 ` fche at redhat dot com
@ 2019-11-24 8:57 ` avi@cloudius-systems.com
2019-12-04 10:32 ` avi@cloudius-systems.com
2019-12-09 22:04 ` fche at redhat dot com
9 siblings, 0 replies; 11+ messages in thread
From: avi@cloudius-systems.com @ 2019-11-24 8:57 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
--- Comment #8 from Avi Kivity <avi@cloudius-systems.com> ---
I imagine that if you notice that a key component is always tid() (except an in
an end probe) then you can rewrite the global map as a thread local map with
extra magic for the end probe.
But it seems fragile, as soon as you violate one of the constraints even a tiny
bit, it stops working with no feedback to the user about what went wrong. And
when it stops working, it's likely to have a huge impact on the running
workload.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
` (7 preceding siblings ...)
2019-11-24 8:57 ` avi@cloudius-systems.com
@ 2019-12-04 10:32 ` avi@cloudius-systems.com
2019-12-09 22:04 ` fche at redhat dot com
9 siblings, 0 replies; 11+ messages in thread
From: avi@cloudius-systems.com @ 2019-12-04 10:32 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
--- Comment #9 from Avi Kivity <avi@cloudius-systems.com> ---
Is there more information I can supply to help fix the segmentation fault?
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Bug dyninst/24884] stapdyn crashes with a segmentation fault
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
` (8 preceding siblings ...)
2019-12-04 10:32 ` avi@cloudius-systems.com
@ 2019-12-09 22:04 ` fche at redhat dot com
9 siblings, 0 replies; 11+ messages in thread
From: fche at redhat dot com @ 2019-12-09 22:04 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=24884
Frank Ch. Eigler <fche at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Assignee|systemtap at sourceware dot org |scox at redhat dot com
--- Comment #10 from Frank Ch. Eigler <fche at redhat dot com> ---
Stan might be able to help with the dyninst segv up in comment #4.
OTOH, there is a dyninst 10.1 build in stable updates, which would be worth
retesting against.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-12-09 22:04 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-05 16:43 [Bug dyninst/24884] New: stapdyn crashes with a segmentation fault avi@cloudius-systems.com
2019-08-23 22:21 ` [Bug dyninst/24884] " fche at redhat dot com
2019-11-12 13:08 ` avi@cloudius-systems.com
2019-11-12 13:11 ` avi@cloudius-systems.com
2019-11-12 13:21 ` avi@cloudius-systems.com
2019-11-12 13:25 ` avi@cloudius-systems.com
2019-11-12 13:38 ` avi@cloudius-systems.com
2019-11-23 1:26 ` fche at redhat dot com
2019-11-24 8:57 ` avi@cloudius-systems.com
2019-12-04 10:32 ` avi@cloudius-systems.com
2019-12-09 22:04 ` fche at redhat dot com
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).