[Bug translator/23866] New: dissonance between kernel tracepoint parametrization, lkm vs bpf

[Bug translator/23866] New: dissonance between kernel tracepoint parametrization, lkm vs bpf

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

            Bug ID: 23866
           Summary: dissonance between kernel tracepoint parametrization,
                    lkm vs bpf
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: translator
          Assignee: systemtap at sourceware dot org
          Reporter: fche at redhat dot com
  Target Milestone: ---

lkm backend tracepoints show the callback-function parameters
bpf backend tracepoints show the saved tracepoint-struct parameters

The former should include the latter.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |wcohen at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
*** Bug 24656 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

--- Comment #2 from William Cohen <wcohen at redhat dot com> ---
Simple example of differences between the lkm and bpf backends target variable:

[wcohen@localhost systemtap]$ ../install/bin/stap -L
'kernel.trace("sched_switch")'
kernel.trace("sched:sched_switch") $preempt:bool $prev:struct task_struct*
$next:struct task_struct*
[wcohen@localhost systemtap]$ ../install/bin/stap --bpf -L
'kernel.trace("sched_switch")'
kernel.trace("sched:sched_switch") $prev_comm:char[] $prev_pid:pid_t
$prev_prio:int $prev_state:long int $next_comm:char[] $next_pid:pid_t
$next_prio:int

The cycle_thief.stp example is affected by this as it uses the
kernel.trace("sched_switch") probe point.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

--- Comment #3 from William Cohen <wcohen at redhat dot com> ---
The original bpf tracepoint support only provided a subset of
predefined list of arguments extracted from the tracepoint arguments.
There are some tracepoints by happenstance that have the same set of
target variables available as the systemtap tracepoint versions of the
probes.  Around March 2018 the kernel added the
BPF_PROG_TYPE_RAW_TRACEPOINT which provides an unprocessed array of
the arguments to the tracepoint (ctx->args[i]).  There are a couple of
iovisor bcc file that describe and use the raw tracepoints:

https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#7-raw-tracepoints

https://github.com/iovisor/bcc/blob/master/tools/runqlat.py
https://github.com/iovisor/bcc/blob/master/tools/runqslower.py

The following blog article provides an example using llvm and bpf
backend to use the bpf raw tracepoint:

https://sysdig.com/blog/the-art-of-writing-ebpf-programs-a-primer/

The above sysdig blog article shows the pointer to the array of 8 byte
entries (ctx->args) is passed in by r1 and outlines how to access the
fields.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

--- Comment #4 from William Cohen <wcohen at redhat dot com> ---
Outline changes to make use of BPF_RAW_TRACEPOINTS

- test and note whether kernel has BPF_PROG_TYPE_RAW_TRACEPOINT available
- reuse the tapsets.cxx for the lkm code to discover the arguments for bpf raw
tracepoints
- modify tapsets.cxx tracepoint_derived_probe::build_args_for_bpf to generate
args for raw_tracepoints
   bpf raw tracepoints have a array of elelements pointed to by r1
- modify the generate tracepoint section names to note when using "raw_trace"
- modify stapbpf.cxx to handle "raw_trace" as another section type

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

--- Comment #5 from William Cohen <wcohen at redhat dot com> ---
Created attachment 11866
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11866&action=edit
WIP patch to generate raw tracepoints

This patch makes switching over to using the raw tracepoints to better match
how the traditional kernel.trace("*") works.  Really simple example:

$ sudo ../install/bin/stap -k --bpf  -m sched -e 'probe
kernel.trace("sched_switch"){printf("hello raw_trace\n");exit()}'
^CWARNING: kbuild exited with signal: 15 (Terminated)
Keeping temporary directory "/tmp/stap1oJtAx"
[wcohen@localhost systemtap]$ sudo ../install/bin/stap -v -k --bpf  -m sched -e
'probe kernel.trace("sched_switch"){printf("hello raw_trace\n");exit()}'
Pass 1: parsed user script and 50 library scripts using
250476virt/23932res/9152shr/14732data kb, in 10usr/10sys/17real ms.
Pass 2: analyzed script: 1 probe, 3 functions, 0 embeds, 0 globals using
256284virt/30968res/10176shr/20540data kb, in 65900usr/28890sys/101380real ms.
Pass 4: compiled BPF into "sched.bo" in 0usr/0sys/13real ms.
Pass 5: starting run.
hello raw_trace
Pass 5: run completed in 0usr/0sys/6real ms.
Keeping temporary directory "/tmp/stapYOYGrA"



This patch still has a number of issues that need to be addressed:

-testing whether raw tracepoints are available on the kernel
-recognizing arguments for tracepoints
-generating bpf to access those arguments

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

--- Comment #6 from William Cohen <wcohen at redhat dot com> ---
Created attachment 11901
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11901&action=edit
Revised version of the patch that gets

This patch makes some changes in the data structure to make it so the same
argument discovery code can be used for both regular and raw tracepoints.
This current version attempt to get the arguments for the various bpf raw
tracepoints.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #11866|0                           |1
        is obsolete|                            |
  Attachment #11901|0                           |1
        is obsolete|                            |

--- Comment #7 from William Cohen <wcohen at redhat dot com> ---
Created attachment 11907
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11907&action=edit
Revised version of the WIP bpf raw tracepoint support

The patch still needs some work:

-modify stapbpf/stapbpf.cxx to allow it to compile on machines that don't have
raw tracepoint support (for example rhel7).
-have clearer handling of things if using the older non-raw bpf tracepoints
-correct access of target struct field arguments (this issue can be seen in the
example below)
-cache tracequery results to speed up the process

$  ../install/bin/stap --bpf -e 'probe kernel.trace("sched_switch"){printf("%d
%p(%d) %p(%d)\n", $preempt, $prev, $prev->pid, $next, $next->pid)}' -T 0.01
0 0xffff8d78706b0000(7260) 0xffff8d78706b3380(7260)
0 0xffff8d78706b3380(29440) 0xffff8d78ba488000(29440)
0 0xffff8d78ba488000(11) 0xffff8d78bcd30000(11)
0 0xffff8d78bcd30000(7260) 0xffff8d78706b3380(7260)
1 0xffff8d78706b3380(1499) 0xffff8d7857d4b380(1499)
0 0xffff8d7857d4b380(6714) 0xffff8d7880628000(6714)
0 0xffff8d7880628000(748) 0xffff8d78ba48b380(748)
0 0xffff8d78ba48b380(29440) 0xffff8d78ba488000(29440)
0 0xffff8d78ba488000(489) 0xffff8d7876410000(489)
0 0xffff8d7876410000(7260) 0xffff8d78706b3380(7260)
0 0xffff8d78706b3380(1499) 0xffff8d7857d4b380(1499)
...

comparing to lkm version which has 0xffff8d78bcd30000(11) looks like getting
the fields for $prev wrong
$  ../install/bin/stap -e 'probe kernel.trace("sched_switch"){printf("%d %p(%d)
%p(%d)\n", $preempt, $prev, $prev->pid, $next, $next->pid)}' -T 0.01
0 0xffff8d78706b0000(11032) 0xffff8d787fb83380(11035)
0 0xffff8d787fb83380(11035) 0xffff8d78bc6b8000(137)
1 0xffff8d78bc6b8000(137) 0xffff8d78ba488000(29440)
0 0xffff8d78ba488000(29440) 0xffff8d78bc6b8000(137)
1 0xffff8d78bc6b8000(137) 0xffff8d78bcd30000(11)
0 0xffff8d78bcd30000(11) 0xffff8d78bc6b8000(137)
1 0xffff8d78bc6b8000(137) 0xffff8d78bcd30000(11)
0 0xffff8d78bcd30000(11) 0xffff8d78bc6b8000(137)
1 0xffff8d78bc6b8000(137) 0xffff8d78bcd30000(11)
0 0xffff8d78bcd30000(11) 0xffff8d78bc6b8000(137)
1 0xffff8d78bc6b8000(137) 0xffff8d7857d48000(10951)
0 0xffff8d7857d48000(10951) 0xffff8d78bc6b8000(137)
1 0xffff8d78bc6b8000(137) 0xffff8d7857d48000(10951)
0 0xffff8d7857d48000(10951) 0xffff8d78bc6b8000(137)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #11907|0                           |1
        is obsolete|                            |

--- Comment #8 from William Cohen <wcohen at redhat dot com> ---
Created attachment 11909
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11909&action=edit
Revised version of the WIP bpf raw tracepoint support

This patch includes configurations machinery to eliminate the stapbfp bpf raw
tracepoints stuff that requires some additional data structure fields and enum
types that are not available on RHEL7.

The access the various fields look reasonable in the following example:

$ uname -a
Linux localhost.localdomain 5.3.0-0.rc0.git4.1.fc31.x86_64 #1 SMP Fri Jul 12
13:34:54 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$  ../install/bin/stap -k -e 'probe kernel.trace("sched_switch"){printf("%d
%s(%d) %s(%d)\n", $preempt, kernel_string($prev->comm), $prev->pid,
kernel_string($next->comm), $next->pid)}' -T 0.01
1 stapio(1505) systemd-journal(576)
0 systemd-journal(576) rcu_sched(11)
0 rcu_sched(11) ksoftirqd/0(10)
0 ksoftirqd/0(10) rcu_sched(11)
0 rcu_sched(11) stapio(1505)
0 stapio(1505) stapio(1508)
0 stapio(1508) swapper/0(0)
0 swapper/0(0) rcu_sched(11)
0 rcu_sched(11) swapper/0(0)
Keeping temporary directory "/tmp/stapto4iHA"
$ more /proc/576/comm 
systemd-journal
$ more /proc/11/comm 
rcu_sched

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #11909|0                           |1
        is obsolete|                            |

--- Comment #9 from William Cohen <wcohen at redhat dot com> ---
Created attachment 11912
  --> https://sourceware.org/bugzilla/attachment.cgi?id=11912&action=edit
Revised version of the WIP bpf raw tracepoint support

Based on feedback previous bpf_raw_tracepoint patch.  This patch has the
following changes:

- NEWS entry
- Made behavior more predictable always use the bpf raw tracepoints unless
--compatible=4.1 used. if bpf raw tracepoints unavailable error out
- Simplified the logic
- Adjusted bpf.exp to work with the new behavior
- Increased bpf.exp timeout so tracepoint discovery has time to run in the
tests

In a separate patch I have a raw_tracepoint1.stp test that is a
reimplementation of the tracepoint1.stp using the bpf raw tracepoints
arguments.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/23866] dissonance between kernel tracepoint parametrization, lkm vs bpf

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=23866

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #10 from William Cohen <wcohen at redhat dot com> ---
This should be addressed by the following commit:

commit fcdd71babea2435601e5e7f8b6b5faae4a663443
Author: William Cohen <wcohen@redhat.com>
Date:   Thu Jul 18 11:37:37 2019 -0400

    PR23866: Make the bpf backend use BPF raw tracepoints for kernel.trace("*")

    The BPF raw tracepoints provide arguments that better match the
    Systemtap lkm kernel tracepoint probes than regular BPF tracepoints.
    The BPF backend will use the BPF raw tracepoints unless the user
    specifies the old behavior with a --compatible=4.1 option on the
    commandline to address.

    The new BPF raw tracepoints and their argument are discovered in the
    same way as the old BPF tracepoints.  A number of small machine
    generated C files are generated with macros are compiled to query the
    available tracepoints in the kernel. Debug information describing the
    data structures passed into the BPF tracepoints is examined to
    determine the type and location of the tracepoint arguments.

    Each probe handler BPF code for BPF raw tracepoints is put into a
    raw_trace section as the method of registering the BPF raw tracepoints
    is different than the regular BPF tracepoints.

    The bpf tests have been revised to include the --compatible=4.1 option
    for the tests where it makes a difference.

-- 
You are receiving this mail because:
You are the assignee for the bug.