[Bug translator/19396] New: systemtap can't find certain kernel tracepoints

[Bug translator/19396] New: systemtap can't find certain kernel tracepoints

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19396

            Bug ID: 19396
           Summary: systemtap can't find certain kernel tracepoints
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: translator
          Assignee: systemtap at sourceware dot org
          Reporter: dsmith at redhat dot com
  Target Milestone: ---

Commit 7649bb8 updated the support for finding kernel tracepoints. However,
even with this commit, systemtap still can't find certain kernel tracepoints.
There is a script, in src/scripts/tracepoint-diff that points this out.

Here are the missing tracepoints, listed by OS.

RHEL6 (2.6.32-573.3.1.el6.x86_64):

module:module_free
module:module_get
module:module_load
module:module_put
module:module_request
napi:napi_poll
net:net_dev_queue
net:net_dev_xmit
net:netif_receive_skb
net:netif_rx
skb:consume_skb
skb:kfree_skb
skb:skb_copy_datagram_iovec
sock:sock_exceed_buf_limit
sock:sock_rcvqueue_full
udp:udp_fail_queue_rcv_skb

RHEL7 (3.10.0-327.el7.x86_64):

ftrace:function
hda:hda_get_response
hda:hda_send_cmd
hda:hda_unsol_event
module:module_free
module:module_get
module:module_load
module:module_put
module:module_request
xfs:* (353 total xfs tracepoints)

Fedora 23 (4.2.7-300.fc23.x86_64):

ftrace:function
hda:hda_get_response
hda:hda_send_cmd
hda:hda_unsol_event
hda:snd_hdac_stream_start
hda:snd_hdac_stream_stop
module:module_free
module:module_get
module:module_load
module:module_put
module:module_request
regmap:regcache_drop_region
regmap:regcache_sync
regmap:regmap_async_complete_done
regmap:regmap_async_complete_start
regmap:regmap_async_io_complete
regmap:regmap_async_write_start
regmap:regmap_cache_bypass
regmap:regmap_cache_only
regmap:regmap_hw_read_done
regmap:regmap_hw_read_start
regmap:regmap_hw_write_done
regmap:regmap_hw_write_start
regmap:regmap_reg_read
regmap:regmap_reg_read_cache
regmap:regmap_reg_write
snd_pcm:hwptr
snd_pcm:hw_ptr_error
snd_pcm:xrun
vsyscall:emulate_vsyscall
xfs:* (359 total xfs tracepoints)

Fedora Rawhide (4.4.0-0.rc4.git4.1.fc24.x86_64):

ftrace:function
hda:hda_get_response
hda:hda_send_cmd
hda:hda_unsol_event
hda:snd_hdac_stream_start
hda:snd_hdac_stream_stop
module:module_free
module:module_get
module:module_load
module:module_put
module:module_request
regmap:regcache_drop_region
regmap:regcache_sync
regmap:regmap_async_complete_done
regmap:regmap_async_complete_start
regmap:regmap_async_io_complete
regmap:regmap_async_write_start
regmap:regmap_cache_bypass
regmap:regmap_cache_only
regmap:regmap_hw_read_done
regmap:regmap_hw_read_start
regmap:regmap_hw_write_done
regmap:regmap_hw_write_start
regmap:regmap_reg_read
regmap:regmap_reg_read_cache
regmap:regmap_reg_write
snd_pcm:hwptr
snd_pcm:hw_ptr_error
snd_pcm:xrun
vsyscall:emulate_vsyscall

We need to convert the 'tracepoint-diff' script to a test case, so we can know
when systemtap can't find a set of tracepoints.

I've debugged this issue somewhat, and the missing xfs tracepoints are because
we need to add a '-I DIR' gcc option to compiles.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/19396] systemtap can't find certain kernel tracepoints

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19396

--- Comment #1 from David Smith <dsmith at redhat dot com> ---
I'm seeing something wacky here. This is on 2.6.32-573.el6.ppc64:

====
# stap --poison-cache -L 'kernel.trace("*module*")'
kernel.trace("jbd:module_free") $mod:struct module*
kernel.trace("jbd:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("jbd:module_load") $mod:struct module*
kernel.trace("jbd:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("jbd:module_request") $name:char* $wait:bool $ip:long unsigned int
====

So, when '--poison-cache' is used, for some reason stap thinks the module_*
tracepoints are in the "jbd" subsystem instead of in the "module" subsystem.
But, you can list them as being in the "module" subsystem:

====
# stap -L 'kernel.trace("*module*")'
kernel.trace("module:module_free") $mod:struct module*
kernel.trace("module:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("module:module_load") $mod:struct module*
kernel.trace("module:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("module:module_request") $name:char* $wait:bool $ip:long unsigned
int
====

You can actually see them in either subsystem:

====
# stap -L 'kernel.trace("jbd:module*")'
kernel.trace("jbd:module_free") $mod:struct module*
kernel.trace("jbd:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("jbd:module_load") $mod:struct module*
kernel.trace("jbd:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("jbd:module_request") $name:char* $wait:bool $ip:long unsigned int
# stap -L 'kernel.trace("module:module*")'
kernel.trace("module:module_free") $mod:struct module*
kernel.trace("module:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("module:module_load") $mod:struct module*
kernel.trace("module:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("module:module_request") $name:char* $wait:bool $ip:long unsigned
int
====

Note that if you try this with a completely non-existing subsystem, you do get
the correct result (of nothing):

====
# stap -L 'kernel.trace("foo:module*")'
#
====

It looks like you can look up the module tracepoints in about any (existing)
subsystem:

====
# stap -L 'kernel.trace("ext4:module*")'
kernel.trace("ext4:module_free") $mod:struct module*
kernel.trace("ext4:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("ext4:module_load") $mod:struct module*
kernel.trace("ext4:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("ext4:module_request") $name:char* $wait:bool $ip:long unsigned
int
# stap -L 'kernel.trace("skb:module*")'
kernel.trace("skb:module_free") $mod:struct module*
kernel.trace("skb:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("skb:module_load") $mod:struct module*
kernel.trace("skb:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("skb:module_request") $name:char* $wait:bool $ip:long unsigned int
# stap -L 'kernel.trace("net:module*")'
kernel.trace("net:module_free") $mod:struct module*
kernel.trace("net:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("net:module_load") $mod:struct module*
kernel.trace("net:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("net:module_request") $name:char* $wait:bool $ip:long unsigned int
# stap -L 'kernel.trace("block:module*")' 
kernel.trace("block:module_free") $mod:struct module*
kernel.trace("block:module_get") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("block:module_load") $mod:struct module*
kernel.trace("block:module_put") $mod:struct module* $ip:long unsigned int
$refcnt:int
kernel.trace("block:module_request") $name:char* $wait:bool $ip:long unsigned
int
====

Also note this isn't just because I used a wildcard, since the following works:

====
# stap -L 'kernel.trace("block:module_free")'
kernel.trace("block:module_free") $mod:struct module*
====

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/19396] systemtap can't find certain kernel tracepoints

[jistone at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19396

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

--- Comment #2 from Josh Stone <jistone at redhat dot com> ---
In Fedora, I only see the module_* tracepoints without any "system:" prefix.
include/trace/events/module.h has:

  #ifdef CREATE_TRACE_POINTS
  #undef TRACE_SYSTEM
  #define TRACE_SYSTEM module
  #endif

AFAICS we don't set CREATE_TRACE_POINTS.  I'm not sure, but maybe we should.

FWIW sched.h and writeback.h also use CREATE_TRACE_POINTS, but only to guard
some helper functions, not to guard TRACE_SYSTEM.  I think that will harmless
to tracequery if those functions are added, as long as they do compile ok.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/19396] systemtap can't find certain kernel tracepoints

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19396

--- Comment #3 from David Smith <dsmith at redhat dot com> ---
From looking at the .o files produced when we're looking for tracepoints, I now
see a bit more at what is going on. For example, looking at
include/trace/events/fence.h, stap builds tracequery_kmod_1_9.c.  Here's the
line from the output of the following command:

stap -k --poison-cache --vp 090 -L 'kernel.trace("*")'

====
Processing tracepoint header
/lib/modules/2.6.32-573.el6.i686/build/include/trace/events/fence.h with query
/tmp/stapi1JkQY/tracequery_kmod_1/tracequery_kmod_1_9.c
====

If I run nm on the resulting .o file, I get:

====
# nm tracequery_kmod_1_9.o
00000000 R stap_trace_system
00000050 T stapprobe_fence_annotate_wait_on
00000080 T stapprobe_fence_destroy
00000060 T stapprobe_fence_emit
00000090 T stapprobe_fence_enable_signal
00000070 T stapprobe_fence_init
000000a0 T stapprobe_fence_signaled
000000c0 T stapprobe_fence_wait_end
000000b0 T stapprobe_fence_wait_start
00000010 T stapprobe_module_free
00000020 T stapprobe_module_get
00000000 T stapprobe_module_load
00000030 T stapprobe_module_put
00000040 T stapprobe_module_request
====

So, the reason why I can look up the module tracepoints as if they were in the
'fence' subsystem is that they are getting included in the fence.h query
module.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/19396] systemtap can't find certain kernel tracepoints

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19396

--- Comment #4 from David Smith <dsmith at redhat dot com> ---
(In reply to Josh Stone from comment #2)
> In Fedora, I only see the module_* tracepoints without any "system:" prefix.
> include/trace/events/module.h has:
> 
>   #ifdef CREATE_TRACE_POINTS
>   #undef TRACE_SYSTEM
>   #define TRACE_SYSTEM module
>   #endif
> 
> AFAICS we don't set CREATE_TRACE_POINTS.  I'm not sure, but maybe we should.
> 
> FWIW sched.h and writeback.h also use CREATE_TRACE_POINTS, but only to guard
> some helper functions, not to guard TRACE_SYSTEM.  I think that will
> harmless to tracequery if those functions are added, as long as they do
> compile ok.

Yeah, I've got a fix for that problem (but haven't checked it in yet). I tried
just defining CREATE_TRACE_POINTS, but that caused other compile problems
later. So, I just hardcoded TRACE_SYSTEM for module.h, which seems to work.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/19396] systemtap can't find certain kernel tracepoints

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19396

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #5 from David Smith <dsmith at redhat dot com> ---
Fixed in commit ac4f283. This commit adds a new testcase that tries to ensure
that systemtap can see every tracepoint that perf sees.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/19396] systemtap can't find certain kernel tracepoints

[jistone at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19396

--- Comment #6 from Josh Stone <jistone at redhat dot com> ---
Did you mean to add stap.log in that commit?  Probably not...

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/19396] systemtap can't find certain kernel tracepoints

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19396

--- Comment #7 from David Smith <dsmith at redhat dot com> ---
(In reply to Josh Stone from comment #6)
> Did you mean to add stap.log in that commit?  Probably not...

Nope, thanks for finding this. Fixed in commit f070ea0.

-- 
You are receiving this mail because:
You are the assignee for the bug.