missing sytemtap kernel.trace("net*") on newer Fedora kernels

missing sytemtap kernel.trace("net*") on newer Fedora kernels

[William Cohen]

I was looking at the perf netdev-times script:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/perf/scripts/python/bin/netdev-times-record

Which uses the following perf tracepoints:

#!/bin/bash
perf record -e net:net_dev_xmit -e net:net_dev_queue		\
		-e net:netif_receive_skb -e net:netif_rx		\
		-e skb:consume_skb -e skb:kfree_skb			\
		-e skb:skb_copy_datagram_iovec -e napi:napi_poll	\
		-e irq:irq_handler_entry -e irq:irq_handler_exit	\
		-e irq:softirq_entry -e irq:softirq_exit		\
		-e irq:softirq_raise $@


On the rhel7 kernel all of the tracepoints are available,  However, on Fedora 20 none of the kernel.trace("net*"), kernel.trace("*skb*"), or kernel.trace("napi_poll") tracepoints are there.

Has anyone else noticed the missing tracepoints on Fedora 20?

-Will

Re: missing sytemtap kernel.trace("net*") on newer Fedora kernels

[Josh Stone]

On 06/11/2014 02:33 PM, William Cohen wrote:
> I was looking at the perf netdev-times script:
> 
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/perf/scripts/python/bin/netdev-times-record
> 
> Which uses the following perf tracepoints:
> 
> #!/bin/bash
> perf record -e net:net_dev_xmit -e net:net_dev_queue		\
> 		-e net:netif_receive_skb -e net:netif_rx		\
> 		-e skb:consume_skb -e skb:kfree_skb			\
> 		-e skb:skb_copy_datagram_iovec -e napi:napi_poll	\
> 		-e irq:irq_handler_entry -e irq:irq_handler_exit	\
> 		-e irq:softirq_entry -e irq:softirq_exit		\
> 		-e irq:softirq_raise $@
> 
> 
> On the rhel7 kernel all of the tracepoints are available,  However,
> on Fedora 20 none of the kernel.trace("net*"), kernel.trace("*skb*"),
> or kernel.trace("napi_poll") tracepoints are there.
> 
> Has anyone else noticed the missing tracepoints on Fedora 20?

I see all of these on kernel-3.14.5-200.fc20.x86_64.

You can see all the gory tracepoint-detecting details from your listing
with "stap -vvvv --poison-cache ...".  For instance, napi_poll should
come from include/trace/events/napi.h.  On my system it finds that
header under both /lib/modules and /usr/src/debug, but I even tried
moving the debug tree aside, and it still found the tracepoint.

Re: missing sytemtap kernel.trace("net*") on newer Fedora kernels

[William Cohen]

On 06/11/2014 07:04 PM, Josh Stone wrote:
> On 06/11/2014 02:33 PM, William Cohen wrote:
>> I was looking at the perf netdev-times script:
>>
>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/tools/perf/scripts/python/bin/netdev-times-record
>>
>> Which uses the following perf tracepoints:
>>
>> #!/bin/bash
>> perf record -e net:net_dev_xmit -e net:net_dev_queue		\
>> 		-e net:netif_receive_skb -e net:netif_rx		\
>> 		-e skb:consume_skb -e skb:kfree_skb			\
>> 		-e skb:skb_copy_datagram_iovec -e napi:napi_poll	\
>> 		-e irq:irq_handler_entry -e irq:irq_handler_exit	\
>> 		-e irq:softirq_entry -e irq:softirq_exit		\
>> 		-e irq:softirq_raise $@
>>
>>
>> On the rhel7 kernel all of the tracepoints are available,  However,
>> on Fedora 20 none of the kernel.trace("net*"), kernel.trace("*skb*"),
>> or kernel.trace("napi_poll") tracepoints are there.
>>
>> Has anyone else noticed the missing tracepoints on Fedora 20?
> 
> I see all of these on kernel-3.14.5-200.fc20.x86_64.
> 
> You can see all the gory tracepoint-detecting details from your listing
> with "stap -vvvv --poison-cache ...".  For instance, napi_poll should
> come from include/trace/events/napi.h.  On my system it finds that
> header under both /lib/modules and /usr/src/debug, but I even tried
> moving the debug tree aside, and it still found the tracepoint.
> 

I upgraded and rebooted to the kernel-3.14.6-200.fc20.x86_64 and the tracepoints seem to be in this kernel. Weird. Now things are working I am going to assume that something got messed on in the kernel-devel install for 3.14.5-200.fc20.x86_64.

-Will

Re: missing sytemtap kernel.trace("net*") on newer Fedora kernels

[Frank Ch. Eigler]

William Cohen <wcohen@redhat.com> writes:

> [...]  On the rhel7 kernel all of the tracepoints are available,
> However, on Fedora 20 none of the kernel.trace("net*"),
> kernel.trace("*skb*"), or kernel.trace("napi_poll") tracepoints are
> there.
> [...]

Those kernel tracepoints that are defined in private headers or .c
files, IOW not included under include/linux/trace/*, require
kernel-debuginfo to be installed for systemtap to find them.  There
are others that are under include/linux/trace/* but the header files
are not self-contained, and thus require stap to emit dummy forward
type declarations.

See also PR9993, the tapsets.cxx tracepoint_extra_decls() function,
and the stap scripts/tracepoint-diff tool.

- FChE