[Bug bpf/25218] New: stapbpf losing perf_events

[Bug bpf/25218] New: stapbpf losing perf_events

[sapatel at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25218

            Bug ID: 25218
           Summary: stapbpf losing perf_events
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: bpf
          Assignee: systemtap at sourceware dot org
          Reporter: sapatel at redhat dot com
  Target Milestone: ---

When there is frequent output from a script, it appears that stapbpf loses some
perf_events which leads to errors. Consider the script below.

global count

probe kernel.function("vfs_read") {

    count++

    printf("Beginning: %d\n", count)
    printf("Middle: %d\n", count)
    printf("End: %d\n", count)

}

Output:

...
...
Beginning: 13700
Middle: 13700
End: 13700
Beginning: 13701
Middle: 13701
End: 13701
WARNING: lost 359 perf_events on cpu 2
bpfinterp.cxx:421: printf already started
WARNING: /home/sapatel/stap_head/install/bin/stapbpf exited with signal: 6
(Aborted)

However, if the output is redirected to a file instead, the perf_events are not
lost, and the script proceeds as expected.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug bpf/25218] stapbpf losing perf_events

[sapatel at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=25218

Sagar Patel <sapatel at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sapatel at redhat dot com

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug bpf/25218] stapbpf loses perf_events when writing to interactive terminal

[me at serhei dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=25218

Serhei Makarov <me at serhei dot io> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |me at serhei dot io
            Summary|stapbpf losing perf_events  |stapbpf loses perf_events
                   |                            |when writing to interactive
                   |                            |terminal

--- Comment #1 from Serhei Makarov <me at serhei dot io> ---
My suspicion is that the perf_events buffer used to receive transport messages
fills up while the stapbpf process is delayed writing output. This results in
transport messages being overwritten and the 'lost perf_events' warning being
generated.

When stapbpf output is redirected to a file, there is no I/O delay, and
perf_events are consumed before the buffer overflows.

If my suspicion is true, this could be solved by receiving transport messages
in a separate thread.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug bpf/25218] stapbpf loses perf_events when writing to interactive terminal

[me at serhei dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=25218

--- Comment #2 from Serhei Makarov <me at serhei dot io> ---
One thing that I should definitely fix, aside from tackling the speed disparity
in console/file output: when perf_events are lost, unpaired PRINTF_{START,END}
messages should not cause a fatal error for stapbpf.

That is, a sequence such as

PRINTF_START PRINTF_END <lost events> PRINTF_END

should cause the PRINTF_END to be dropped along with the lost events.
Currently it causes stapbpf to terminate with an error message.

-- 
You are receiving this mail because:
You are the assignee for the bug.