[Bug bpf/24324] New: bpf map entry 0: Operation not permitted

[Bug bpf/24324] New: bpf map entry 0: Operation not permitted

[me at serhei dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=24324

            Bug ID: 24324
           Summary: bpf map entry 0: Operation not permitted
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: bpf
          Assignee: systemtap at sourceware dot org
          Reporter: me at serhei dot io
  Target Milestone: ---

This is an annoying issue that's related to ulimit. After a long time running
different stapbpf programs, previously working scripts begin to fail, being
unable to allocate maps.

(On older Fedora kernels I saw an issue where instead the BPF verifier would
reject a program -- that seems unrelated and difficult to reproduce.)

The issue can be worked around by removing RLIMIT_MEMLOCK entirely in
stapbpf.cxx instantiate_maps():

+  curr_rlimit.rlim_cur = RLIM_INFINITY;
+  curr_rlimit.rlim_max = RLIM_INFINITY;
   rc = setrlimit(RLIMIT_MEMLOCK, &curr_rlimit);

This is also what bcc does. However, the fact that rlimit resources do not
overflow immediately, but are instead exhausted over multiple runs and across a
reboot makes me suspicious about the safety of this fix.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug bpf/24324] bpf map entry 0: Operation not permitted

[me at serhei dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=24324

Serhei Makarov <me at serhei dot io> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |critical

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug bpf/24324] bpf map entry 0: Operation not permitted

[me at serhei dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=24324

--- Comment #1 from Serhei Makarov <me at serhei dot io> ---
As suspected, applying this quick fix on Fedora can lead to subtler resource
exhaustion, so committing it to master is a no-go for now.

I do notice that the testsuite can produce orphaned stapbpf processes which may
be holding the pinned memory, or some other resource that runs out more slowly
than the finite rlimit I was using.

root       482  0.0  0.0  80144  3848 pts/6    D    16:18   0:00
/opt/systemtap/bin/stapbpf /tmp/stapfEnTr4/stap_481.bo
root       494  0.0  0.0   6412  1760 pts/6    D    16:18   0:00
/opt/systemtap/bin/stapbpf /tmp/stapm3M5Nd/stap_493.bo
root       508  0.0  0.0   6376  1676 pts/6    D    16:19   0:00
/opt/systemtap/bin/stapbpf /tmp/stap7RZrMT/stap_507.bo
root       643  0.0  0.0   6384  1720 pts/6    D    16:21   0:00
/opt/systemtap/bin/stapbpf /tmp/stapMhTLZX/stap_642.bo
root       666  0.0  0.0   6384  1644 pts/6    D    16:21   0:00
/opt/systemtap/bin/stapbpf /tmp/stap1ME9en/stap_663.bo
root       688  0.0  0.0   6384  1648 pts/6    D    16:22   0:00
/opt/systemtap/bin/stapbpf /tmp/stapd3AOZs/stap_687.bo
serhei     746  0.0  0.0   6004   760 pts/6    S+   16:22   0:00 grep
--color=auto stapbpf

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug bpf/24324] bpf map entry 0: Operation not permitted

[wcohen at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=24324

William Cohen <wcohen at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |wcohen at redhat dot com

--- Comment #2 from William Cohen <wcohen at redhat dot com> ---
When shutting down bpf instrumentation what happens to the map elements
created?
Are they automatically remove when the ebpf program that created them goes
away?
Or do the maps need to be explicitly cleaned up?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug bpf/24324] bpf map entry 0: Operation not permitted

[me at serhei dot io]

https://sourceware.org/bugzilla/show_bug.cgi?id=24324

--- Comment #3 from Serhei Makarov <me at serhei dot io> ---
IIRC by default bpf maps, programs, and so forth will be garbage collected when
the stapbpf process that owns them is stopped.

However, as documented in my previous comment, getting the stapbpf processes to
always stop cleanly is the real problem.

-- 
You are receiving this mail because:
You are the assignee for the bug.