* [Bug translator/19396] New: systemtap can't find certain kernel tracepoints
@ 2015-12-22 21:37 dsmith at redhat dot com
2016-01-06 17:12 ` [Bug translator/19396] " dsmith at redhat dot com
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: dsmith at redhat dot com @ 2015-12-22 21:37 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=19396
Bug ID: 19396
Summary: systemtap can't find certain kernel tracepoints
Product: systemtap
Version: unspecified
Status: NEW
Severity: normal
Priority: P2
Component: translator
Assignee: systemtap at sourceware dot org
Reporter: dsmith at redhat dot com
Target Milestone: ---
Commit 7649bb8 updated the support for finding kernel tracepoints. However,
even with this commit, systemtap still can't find certain kernel tracepoints.
There is a script, in src/scripts/tracepoint-diff that points this out.
Here are the missing tracepoints, listed by OS.
RHEL6 (2.6.32-573.3.1.el6.x86_64):
module:module_free
module:module_get
module:module_load
module:module_put
module:module_request
napi:napi_poll
net:net_dev_queue
net:net_dev_xmit
net:netif_receive_skb
net:netif_rx
skb:consume_skb
skb:kfree_skb
skb:skb_copy_datagram_iovec
sock:sock_exceed_buf_limit
sock:sock_rcvqueue_full
udp:udp_fail_queue_rcv_skb
RHEL7 (3.10.0-327.el7.x86_64):
ftrace:function
hda:hda_get_response
hda:hda_send_cmd
hda:hda_unsol_event
module:module_free
module:module_get
module:module_load
module:module_put
module:module_request
xfs:* (353 total xfs tracepoints)
Fedora 23 (4.2.7-300.fc23.x86_64):
ftrace:function
hda:hda_get_response
hda:hda_send_cmd
hda:hda_unsol_event
hda:snd_hdac_stream_start
hda:snd_hdac_stream_stop
module:module_free
module:module_get
module:module_load
module:module_put
module:module_request
regmap:regcache_drop_region
regmap:regcache_sync
regmap:regmap_async_complete_done
regmap:regmap_async_complete_start
regmap:regmap_async_io_complete
regmap:regmap_async_write_start
regmap:regmap_cache_bypass
regmap:regmap_cache_only
regmap:regmap_hw_read_done
regmap:regmap_hw_read_start
regmap:regmap_hw_write_done
regmap:regmap_hw_write_start
regmap:regmap_reg_read
regmap:regmap_reg_read_cache
regmap:regmap_reg_write
snd_pcm:hwptr
snd_pcm:hw_ptr_error
snd_pcm:xrun
vsyscall:emulate_vsyscall
xfs:* (359 total xfs tracepoints)
Fedora Rawhide (4.4.0-0.rc4.git4.1.fc24.x86_64):
ftrace:function
hda:hda_get_response
hda:hda_send_cmd
hda:hda_unsol_event
hda:snd_hdac_stream_start
hda:snd_hdac_stream_stop
module:module_free
module:module_get
module:module_load
module:module_put
module:module_request
regmap:regcache_drop_region
regmap:regcache_sync
regmap:regmap_async_complete_done
regmap:regmap_async_complete_start
regmap:regmap_async_io_complete
regmap:regmap_async_write_start
regmap:regmap_cache_bypass
regmap:regmap_cache_only
regmap:regmap_hw_read_done
regmap:regmap_hw_read_start
regmap:regmap_hw_write_done
regmap:regmap_hw_write_start
regmap:regmap_reg_read
regmap:regmap_reg_read_cache
regmap:regmap_reg_write
snd_pcm:hwptr
snd_pcm:hw_ptr_error
snd_pcm:xrun
vsyscall:emulate_vsyscall
We need to convert the 'tracepoint-diff' script to a test case, so we can know
when systemtap can't find a set of tracepoints.
I've debugged this issue somewhat, and the missing xfs tracepoints are because
we need to add a '-I DIR' gcc option to compiles.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Bug translator/19396] systemtap can't find certain kernel tracepoints
2015-12-22 21:37 [Bug translator/19396] New: systemtap can't find certain kernel tracepoints dsmith at redhat dot com
@ 2016-01-06 17:12 ` dsmith at redhat dot com
2016-01-06 20:40 ` jistone at redhat dot com
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: dsmith at redhat dot com @ 2016-01-06 17:12 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=19396
--- Comment #1 from David Smith <dsmith at redhat dot com> ---
I'm seeing something wacky here. This is on 2.6.32-573.el6.ppc64:
====
# stap --poison-cache -L 'kernel.trace("*module*")'
kernel.trace("jbd:module_free") $mod:struct module*
kernel.trace("jbd:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("jbd:module_load") $mod:struct module*
kernel.trace("jbd:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("jbd:module_request") $name:char* $wait:bool $ip:long unsigned int
====
So, when '--poison-cache' is used, for some reason stap thinks the module_*
tracepoints are in the "jbd" subsystem instead of in the "module" subsystem.
But, you can list them as being in the "module" subsystem:
====
# stap -L 'kernel.trace("*module*")'
kernel.trace("module:module_free") $mod:struct module*
kernel.trace("module:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("module:module_load") $mod:struct module*
kernel.trace("module:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("module:module_request") $name:char* $wait:bool $ip:long unsigned
int
====
You can actually see them in either subsystem:
====
# stap -L 'kernel.trace("jbd:module*")'
kernel.trace("jbd:module_free") $mod:struct module*
kernel.trace("jbd:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("jbd:module_load") $mod:struct module*
kernel.trace("jbd:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("jbd:module_request") $name:char* $wait:bool $ip:long unsigned int
# stap -L 'kernel.trace("module:module*")'
kernel.trace("module:module_free") $mod:struct module*
kernel.trace("module:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("module:module_load") $mod:struct module*
kernel.trace("module:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("module:module_request") $name:char* $wait:bool $ip:long unsigned
int
====
Note that if you try this with a completely non-existing subsystem, you do get
the correct result (of nothing):
====
# stap -L 'kernel.trace("foo:module*")'
#
====
It looks like you can look up the module tracepoints in about any (existing)
subsystem:
====
# stap -L 'kernel.trace("ext4:module*")'
kernel.trace("ext4:module_free") $mod:struct module*
kernel.trace("ext4:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("ext4:module_load") $mod:struct module*
kernel.trace("ext4:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("ext4:module_request") $name:char* $wait:bool $ip:long unsigned
int
# stap -L 'kernel.trace("skb:module*")'
kernel.trace("skb:module_free") $mod:struct module*
kernel.trace("skb:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("skb:module_load") $mod:struct module*
kernel.trace("skb:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("skb:module_request") $name:char* $wait:bool $ip:long unsigned int
# stap -L 'kernel.trace("net:module*")'
kernel.trace("net:module_free") $mod:struct module*
kernel.trace("net:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("net:module_load") $mod:struct module*
kernel.trace("net:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("net:module_request") $name:char* $wait:bool $ip:long unsigned int
# stap -L 'kernel.trace("block:module*")'
kernel.trace("block:module_free") $mod:struct module*
kernel.trace("block:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("block:module_load") $mod:struct module*
kernel.trace("block:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("block:module_request") $name:char* $wait:bool $ip:long unsigned
int
====
Also note this isn't just because I used a wildcard, since the following works:
====
# stap -L 'kernel.trace("block:module_free")'
kernel.trace("block:module_free") $mod:struct module*
====
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Bug translator/19396] systemtap can't find certain kernel tracepoints
2015-12-22 21:37 [Bug translator/19396] New: systemtap can't find certain kernel tracepoints dsmith at redhat dot com
2016-01-06 17:12 ` [Bug translator/19396] " dsmith at redhat dot com
@ 2016-01-06 20:40 ` jistone at redhat dot com
2016-01-06 21:55 ` dsmith at redhat dot com
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: jistone at redhat dot com @ 2016-01-06 20:40 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=19396
Josh Stone <jistone at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jistone at redhat dot com
--- Comment #2 from Josh Stone <jistone at redhat dot com> ---
In Fedora, I only see the module_* tracepoints without any "system:" prefix.
include/trace/events/module.h has:
#ifdef CREATE_TRACE_POINTS
#undef TRACE_SYSTEM
#define TRACE_SYSTEM module
#endif
AFAICS we don't set CREATE_TRACE_POINTS. I'm not sure, but maybe we should.
FWIW sched.h and writeback.h also use CREATE_TRACE_POINTS, but only to guard
some helper functions, not to guard TRACE_SYSTEM. I think that will harmless
to tracequery if those functions are added, as long as they do compile ok.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Bug translator/19396] systemtap can't find certain kernel tracepoints
2015-12-22 21:37 [Bug translator/19396] New: systemtap can't find certain kernel tracepoints dsmith at redhat dot com
2016-01-06 17:12 ` [Bug translator/19396] " dsmith at redhat dot com
2016-01-06 20:40 ` jistone at redhat dot com
@ 2016-01-06 21:55 ` dsmith at redhat dot com
2016-01-06 21:57 ` dsmith at redhat dot com
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: dsmith at redhat dot com @ 2016-01-06 21:55 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=19396
--- Comment #3 from David Smith <dsmith at redhat dot com> ---
From looking at the .o files produced when we're looking for tracepoints, I now
see a bit more at what is going on. For example, looking at
include/trace/events/fence.h, stap builds tracequery_kmod_1_9.c. Here's the
line from the output of the following command:
stap -k --poison-cache --vp 090 -L 'kernel.trace("*")'
====
Processing tracepoint header
/lib/modules/2.6.32-573.el6.i686/build/include/trace/events/fence.h with query
/tmp/stapi1JkQY/tracequery_kmod_1/tracequery_kmod_1_9.c
====
If I run nm on the resulting .o file, I get:
====
# nm tracequery_kmod_1_9.o
00000000 R stap_trace_system
00000050 T stapprobe_fence_annotate_wait_on
00000080 T stapprobe_fence_destroy
00000060 T stapprobe_fence_emit
00000090 T stapprobe_fence_enable_signal
00000070 T stapprobe_fence_init
000000a0 T stapprobe_fence_signaled
000000c0 T stapprobe_fence_wait_end
000000b0 T stapprobe_fence_wait_start
00000010 T stapprobe_module_free
00000020 T stapprobe_module_get
00000000 T stapprobe_module_load
00000030 T stapprobe_module_put
00000040 T stapprobe_module_request
====
So, the reason why I can look up the module tracepoints as if they were in the
'fence' subsystem is that they are getting included in the fence.h query
module.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Bug translator/19396] systemtap can't find certain kernel tracepoints
2015-12-22 21:37 [Bug translator/19396] New: systemtap can't find certain kernel tracepoints dsmith at redhat dot com
` (2 preceding siblings ...)
2016-01-06 21:55 ` dsmith at redhat dot com
@ 2016-01-06 21:57 ` dsmith at redhat dot com
2016-01-08 21:46 ` dsmith at redhat dot com
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: dsmith at redhat dot com @ 2016-01-06 21:57 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=19396
--- Comment #4 from David Smith <dsmith at redhat dot com> ---
(In reply to Josh Stone from comment #2)
> In Fedora, I only see the module_* tracepoints without any "system:" prefix.
> include/trace/events/module.h has:
>
> #ifdef CREATE_TRACE_POINTS
> #undef TRACE_SYSTEM
> #define TRACE_SYSTEM module
> #endif
>
> AFAICS we don't set CREATE_TRACE_POINTS. I'm not sure, but maybe we should.
>
> FWIW sched.h and writeback.h also use CREATE_TRACE_POINTS, but only to guard
> some helper functions, not to guard TRACE_SYSTEM. I think that will
> harmless to tracequery if those functions are added, as long as they do
> compile ok.
Yeah, I've got a fix for that problem (but haven't checked it in yet). I tried
just defining CREATE_TRACE_POINTS, but that caused other compile problems
later. So, I just hardcoded TRACE_SYSTEM for module.h, which seems to work.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Bug translator/19396] systemtap can't find certain kernel tracepoints
2015-12-22 21:37 [Bug translator/19396] New: systemtap can't find certain kernel tracepoints dsmith at redhat dot com
` (3 preceding siblings ...)
2016-01-06 21:57 ` dsmith at redhat dot com
@ 2016-01-08 21:46 ` dsmith at redhat dot com
2016-01-08 23:16 ` jistone at redhat dot com
2016-01-11 13:51 ` dsmith at redhat dot com
6 siblings, 0 replies; 8+ messages in thread
From: dsmith at redhat dot com @ 2016-01-08 21:46 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=19396
David Smith <dsmith at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #5 from David Smith <dsmith at redhat dot com> ---
Fixed in commit ac4f283. This commit adds a new testcase that tries to ensure
that systemtap can see every tracepoint that perf sees.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Bug translator/19396] systemtap can't find certain kernel tracepoints
2015-12-22 21:37 [Bug translator/19396] New: systemtap can't find certain kernel tracepoints dsmith at redhat dot com
` (4 preceding siblings ...)
2016-01-08 21:46 ` dsmith at redhat dot com
@ 2016-01-08 23:16 ` jistone at redhat dot com
2016-01-11 13:51 ` dsmith at redhat dot com
6 siblings, 0 replies; 8+ messages in thread
From: jistone at redhat dot com @ 2016-01-08 23:16 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=19396
--- Comment #6 from Josh Stone <jistone at redhat dot com> ---
Did you mean to add stap.log in that commit? Probably not...
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Bug translator/19396] systemtap can't find certain kernel tracepoints
2015-12-22 21:37 [Bug translator/19396] New: systemtap can't find certain kernel tracepoints dsmith at redhat dot com
` (5 preceding siblings ...)
2016-01-08 23:16 ` jistone at redhat dot com
@ 2016-01-11 13:51 ` dsmith at redhat dot com
6 siblings, 0 replies; 8+ messages in thread
From: dsmith at redhat dot com @ 2016-01-11 13:51 UTC (permalink / raw)
To: systemtap
https://sourceware.org/bugzilla/show_bug.cgi?id=19396
--- Comment #7 from David Smith <dsmith at redhat dot com> ---
(In reply to Josh Stone from comment #6)
> Did you mean to add stap.log in that commit? Probably not...
Nope, thanks for finding this. Fixed in commit f070ea0.
--
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-01-11 13:51 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-22 21:37 [Bug translator/19396] New: systemtap can't find certain kernel tracepoints dsmith at redhat dot com
2016-01-06 17:12 ` [Bug translator/19396] " dsmith at redhat dot com
2016-01-06 20:40 ` jistone at redhat dot com
2016-01-06 21:55 ` dsmith at redhat dot com
2016-01-06 21:57 ` dsmith at redhat dot com
2016-01-08 21:46 ` dsmith at redhat dot com
2016-01-08 23:16 ` jistone at redhat dot com
2016-01-11 13:51 ` dsmith at redhat dot com
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).