From: "awillia2 at sourcefire dot com" <sourceware-bugzilla@sourceware.org>
To: systemtap@sourceware.org
Subject: [Bug kprobes/23260] SystemTap usage appears to adversely affect GDB breakpoint functionality
Date: Fri, 08 Jun 2018 22:01:00 -0000 [thread overview]
Message-ID: <bug-23260-6586-OkgMVN7C4H@http.sourceware.org/bugzilla/> (raw)
In-Reply-To: <bug-23260-6586@http.sourceware.org/bugzilla/>
https://sourceware.org/bugzilla/show_bug.cgi?id=23260
--- Comment #3 from awillia2 at sourcefire dot com ---
I wasn't able to get the stap2perf working correctly with the strace.stp
script, but I did run "sudo perf record -e 'syscalls:sys_enter_*' -p
<hello-world pid>" while gdb was stopped at the breakpoint, and running
'continue' afterward, the hello-world program finished successfully. Also, I
confirmed that running stap with -DALIBI did not prevent the issue from
occurring.
I did some more debugging with
https://sourceware.org/systemtap/examples/process/sigmon.stp (with the if
condition removed) and also found a way to reproduce the second issue described
above (using a 64-bit build of the hello-world program this time):
$ gdb ./hello-world
GNU gdb (Ubuntu 8.0.1-0ubuntu1) 8.0.1
[ ... ]
(gdb) info files
Symbols from "/home/zelda/Desktop/hello-world".
Local exec file:
`/home/zelda/Desktop/hello-world', file type elf64-x86-64.
Entry point: 0x44e370
0x0000000000401000 - 0x000000000047b8ab is .text
0x000000000047c000 - 0x00000000004afd75 is .rodata
0x00000000004afea0 - 0x00000000004b09dc is .typelink
0x00000000004b09e0 - 0x00000000004b0a18 is .itablink
0x00000000004b0a18 - 0x00000000004b0a18 is .gosymtab
0x00000000004b0a20 - 0x00000000004f53b3 is .gopclntab
0x00000000004f6000 - 0x00000000004f84c8 is .noptrdata
0x00000000004f84e0 - 0x00000000004fa1f0 is .data
0x00000000004fa200 - 0x0000000000514b08 is .bss
0x0000000000514b20 - 0x00000000005191c0 is .noptrbss
0x0000000000400fc8 - 0x0000000000401000 is .note.go.buildid
(gdb) break *0x44e370
Breakpoint 1 at 0x44e370: file /usr/lib/go-1.8/src/runtime/rt0_linux_amd64.s,
line 8.
(gdb) run
Starting program: /home/zelda/Desktop/hello-world
Breakpoint 1, _rt0_amd64_linux ()
at /usr/lib/go-1.8/src/runtime/rt0_linux_amd64.s:8
8 LEAQ 8(SP), SI // argv
(gdb)
[ In a second terminal: ]
$ ps -eL -o pid,ppid,lwp,stat,nlwp,cmd | awk 'NR==1 ||
/\/home\/zelda\/Desktop\/hello-world/'
PID PPID LWP STAT NLWP CMD
49663 49118 49663 t 1 /home/zelda/Desktop/hello-world
[ In a third terminal: ]
$ sudo stap --suppress-handler-errors --skip-badvars strace.stp -x 49663 -v
Pass 1: parsed user script and 465 library scripts using
119616virt/48792res/6676shr/42428data kb, in 100usr/50sys/181real ms.
Pass 2: analyzed script: 1116 probes, 207 functions, 104 embeds, 108 globals
using 311348virt/241716res/7876shr/234160data kb, in 23180usr/850sys/25056real
ms.
Pass 3: using cached
/home/zelda/.systemtap/cache/02/stap_027e862ab47a5cdf6a07ae0b1c596ae7_1079724.c
Pass 4: using cached
/home/zelda/.systemtap/cache/02/stap_027e862ab47a5cdf6a07ae0b1c596ae7_1079724.ko
Pass 5: starting run.
Fri Jun 8 20:59:17 2018.167252 arch_prctl(ARCH_SET_FS, 5222056) = 0
Fri Jun 8 20:59:17 2018.168042 sched_getaffinity(0, 8192, 0x7fffffffc078) = 16
Fri Jun 8 20:59:17 2018.168409 mmap2(0xc000000000, 65536, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000
Fri Jun 8 20:59:17 2018.168421 munmap(0xc000000000, 65536) = 0
Fri Jun 8 20:59:17 2018.168445 mmap2(0x0, 262144, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffff7fba000
Fri Jun 8 20:59:17 2018.168604 mmap2(0xc420000000, 1048576,
PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc420000000
Fri Jun 8 20:59:17 2018.168613 mmap2(0xc41fff8000, 32768,
PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc41fff8000
Fri Jun 8 20:59:17 2018.168618 mmap2(0xc000000000, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xc000000000
Fri Jun 8 20:59:17 2018.168622 mmap2(0x0, 65536, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffff7faa000
Fri Jun 8 20:59:17 2018.168642 mmap2(0x0, 65536, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffff7f9a000
Fri Jun 8 20:59:17 2018.168923 rt_sigprocmask(SIG_SETMASK, NULL, 0x4fae98, 8)
= 0
Fri Jun 8 20:59:17 2018.169039 sigaltstack(0x0, 0x7fffffffe010) = 0
Fri Jun 8 20:59:17 2018.169046 sigaltstack({ss_sp=0xc420002000, ss_flags=0x0,
ss_size=32672}, 0x0) = 0
Fri Jun 8 20:59:17 2018.169052 rt_sigprocmask(SIG_SETMASK, [EMPTY], 0x0, 8) =
0
Fri Jun 8 20:59:17 2018.169056 gettid() = 49663
Fri Jun 8 20:59:17 2018.169063 rt_sigaction(SIGHUP, {NULL}, 0x7fffffffdfe8, 8)
= 0
Fri Jun 8 20:59:17 2018.169081 rt_sigaction(SIGHUP, {0x44e6f0,
SA_ONSTACK|SA_RESTART|SA_SIGINFO|SA_RESTORER, 0x44e820,
[SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGBUS|SIGFPE|SIGKILL|SIGUSR1|SIGSEGV|SIGPIPE|SIGUSR2|SIGALRM|SIGTERM|SIGCHLD|SIGCONT|SIGSTOP|SIGTSTP|SIGTTIN|SIGTTOU|SIGURG|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGIO/SIGPOLL|SIGPWR|SIGSYS]},
0x0, 8) = 0
[ ... ]
Fri Jun 8 20:59:17 2018.169864 rt_sigaction(0x40, {NULL}, 0x7fffffffdfe8, 8) =
0
Fri Jun 8 20:59:17 2018.169874 rt_sigaction(0x40, {0x44e6f0,
SA_ONSTACK|SA_RESTART|SA_SIGINFO|SA_RESTORER, 0x44e820,
[SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGBUS|SIGFPE|SIGKILL|SIGUSR1|SIGSEGV|SIGPIPE|SIGUSR2|SIGALRM|SIGTERM|SIGCHLD|SIGCONT|SIGSTOP|SIGTSTP|SIGTTIN|SIGTTOU|SIGURG|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGIO/SIGPOLL|SIGPWR|SIGSYS]},
0x0, 8) = 0
Fri Jun 8 20:59:17 2018.169897 rt_sigprocmask(SIG_SETMASK,
[SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGBUS|SIGFPE|SIGKILL|SIGUSR1|SIGSEGV|SIGPIPE|SIGUSR2|SIGALRM|SIGTERM|SIGCHLD|SIGCONT|SIGSTOP|SIGTSTP|SIGTTIN|SIGTTOU|SIGURG|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGIO/SIGPOLL|SIGPWR|SIGSYS],
0x7fffffffdfb8, 8) = 0
[ Printed by sigmon.stp when strace.stp starts: ]
SPID SNAME RPID RNAME SIGNUM SIGNAME
...
49663 hello-world 49118 gdb 17 SIGCHLD
49663 hello-world 49118 gdb 17 SIGCHLD
[ In the second terminal, notice the new LWP: ]
$ ps -eL -o pid,ppid,lwp,stat,nlwp,cmd | awk 'NR==1 ||
/\/home\/zelda\/Desktop\/hello-world/'
PID PPID LWP STAT NLWP CMD
49663 49118 49663 tl 2 /home/zelda/Desktop/hello-world
49663 49118 49672 tl 2 /home/zelda/Desktop/hello-world
[ In the third terminal, Control-C the running stap and rerun: ]
$ sudo stap --suppress-handler-errors --skip-badvars strace.stp -x 49663 -v
Pass 1: parsed user script and 465 library scripts using
119616virt/48708res/6588shr/42428data kb, in 90usr/50sys/180real ms.
Pass 2: analyzed script: 1116 probes, 207 functions, 104 embeds, 108 globals
using 311348virt/241568res/7720shr/234160data kb, in 21960usr/880sys/24094real
ms.
Pass 3: using cached
/home/zelda/.systemtap/cache/02/stap_027e862ab47a5cdf6a07ae0b1c596ae7_1079724.c
Pass 4: using cached
/home/zelda/.systemtap/cache/02/stap_027e862ab47a5cdf6a07ae0b1c596ae7_1079724.ko
Pass 5: starting run.
Fri Jun 8 21:01:52 2018.772129 rt_sigprocmask(SIG_SETMASK, [EMPTY], 0x0, 8) =
0
Fri Jun 8 21:01:52 2018.773622 rt_sigprocmask(SIG_SETMASK,
[SIGHUP|SIGINT|SIGQUIT|SIGILL|SIGTRAP|SIGABRT|SIGBUS|SIGFPE|SIGKILL|SIGUSR1|SIGSEGV|SIGPIPE|SIGUSR2|SIGALRM|SIGTERM|SIGCHLD|SIGCONT|SIGSTOP|SIGTSTP|SIGTTIN|SIGTTOU|SIGURG|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGIO/SIGPOLL|SIGPWR|SIGSYS],
0x7fffffffdee8, 8) = 0
[ Printed again by sigmon.stp: ]
SPID SNAME RPID RNAME SIGNUM SIGNAME
...
49663 hello-world 49118 gdb 17 SIGCHLD
49663 hello-world 49118 gdb 17 SIGCHLD
[ In the second terminal, notice the new LWP: ]
$ ps -eL -o pid,ppid,lwp,stat,nlwp,cmd | awk 'NR==1 ||
/\/home\/zelda\/Desktop\/hello-world/'
PID PPID LWP STAT NLWP CMD
49663 49118 49663 tl 3 /home/zelda/Desktop/hello-world
49663 49118 49672 tl 3 /home/zelda/Desktop/hello-world
49663 49118 49691 tl 3 /home/zelda/Desktop/hello-world
[ In the third terminal, Control-C the running stap and rerun: ]
zelda@hyrule:~/Desktop$ sudo stap --suppress-handler-errors --skip-badvars
strace.stp -x 49663 -v
Pass 1: parsed user script and 465 library scripts using
119612virt/48624res/6504shr/42424data kb, in 90usr/60sys/181real ms.
Pass 2: analyzed script: 1116 probes, 207 functions, 104 embeds, 108 globals
using 311344virt/241252res/7408shr/234156data kb, in 22270usr/910sys/27506real
ms.
Pass 3: using cached
/home/zelda/.systemtap/cache/02/stap_027e862ab47a5cdf6a07ae0b1c596ae7_1079724.c
Pass 4: using cached
/home/zelda/.systemtap/cache/02/stap_027e862ab47a5cdf6a07ae0b1c596ae7_1079724.ko
Pass 5: starting run.
Fri Jun 8 21:04:54 2018.291180 rt_sigprocmask(SIG_SETMASK, [EMPTY], 0x0, 8) =
0
[ sigmon.stp indicates that no signals were sent to GDB ]
[ In the second terminal, notice the STAT change to the first process: ]
$ ps -eL -o pid,ppid,lwp,stat,nlwp,cmd | awk 'NR==1 ||
/\/home\/zelda\/Desktop\/hello-world/'
PID PPID LWP STAT NLWP CMD
49663 49118 49663 Sl 3 /home/zelda/Desktop/hello-world
49663 49118 49672 tl 3 /home/zelda/Desktop/hello-world
49663 49118 49691 tl 3 /home/zelda/Desktop/hello-world
[ In the original terminal running gdb: ]
(gdb) c
Continuing.
Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0x44e370
Command aborted.
(gdb)
One other thing worth mentioning is that I ran strace.stp against the gdb
process itself as well, and when stopped at the breakpoint and strace.stp runs
for the first time against the child process, the following output is observed:
Fri Jun 8 19:22:23 2018.502985 rt_sigreturn() = -4 (EINTR)
Fri Jun 8 19:22:23 2018.502995 rt_sigreturn() = -4 (EINTR)
Fri Jun 8 19:22:23 2018.503581 read(5, 0x7fff65f9e207, 1) = -11 (EAGAIN)
Fri Jun 8 19:22:23 2018.503781 poll(0x56433c570cb0, 4, 0) = 0
Fri Jun 8 19:22:23 2018.503793 poll(0x56433c570cb0, 4, -1) = 1
This seems odd, since the rt_sigreturn man page indicates that the function
doesn't return.
--
You are receiving this mail because:
You are the assignee for the bug.
next prev parent reply other threads:[~2018-06-08 22:01 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-05 18:06 [Bug kprobes/23260] New: " awillia2 at sourcefire dot com
2018-06-05 18:08 ` [Bug kprobes/23260] " awillia2 at sourcefire dot com
2018-06-06 17:58 ` fche at redhat dot com
2018-06-08 22:01 ` awillia2 at sourcefire dot com [this message]
2018-07-19 15:59 ` awillia2 at sourcefire dot com
2018-07-25 20:50 ` fche at redhat dot com
2020-02-19 21:10 ` fche at redhat dot com
2020-07-09 18:34 ` amerey at redhat dot com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bug-23260-6586-OkgMVN7C4H@http.sourceware.org/bugzilla/ \
--to=sourceware-bugzilla@sourceware.org \
--cc=systemtap@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).