[Bug runtime/15805] New: maps should not be allocated with k*alloc

[Bug runtime/15805] New: maps should not be allocated with k*alloc

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15805

            Bug ID: 15805
           Summary: maps should not be allocated with k*alloc
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: fche at redhat dot com

We should (re)turn(?) to using vmalloc to get the potentially
multi-megabyte-sized chunks of memory used for runtime/map objects.
It's too easy for kernel free memory to be fragmented, and kmalloc
to fail, even for moderate sized objects.

The attached OOM was seen in the wild for a 3.9.4-200.fc18.x86_64
kernel, running a script with 

global foo
probe ... { foo[pid(),tid()]=$number }

i.e., only about a hundred kilobytes' worth.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15805] maps should not be allocated with k*alloc

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15805

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
Created attachment 7128
  --> http://sourceware.org/bugzilla/attachment.cgi?id=7128&action=edit
oom message

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15805] maps should not be allocated with k*alloc

[jkaluza at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15805

Jan Kaluza <jkaluza at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jkaluza at redhat dot com

--- Comment #2 from Jan Kaluza <jkaluza at redhat dot com> ---
I'm the original reproducer of this bug and I presume I'll be able to reproduce
it for some time if you want some tests or information.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15805] maps should not be allocated with k*alloc

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15805

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #3 from David Smith <dsmith at redhat dot com> ---
Created attachment 7135
  --> http://sourceware.org/bugzilla/attachment.cgi?id=7135&action=edit
potential patch

Here's a patch that changes map allocation to use vmalloc instead of kmalloc.
Tested on rhel5 (ia64, x86_64) and rawhide (x86_64).

Note a couple of things:

1) There isn't a exported rawhide kernel vmalloc variant that supports
allocating on a particular node and lets you set flags. I chose to use the
variant that lets you allocate on a particular node.

2) On rhel5 ia64 (2.6.18-348.6.1.el5debug), I'm seeing some bad/odd behavior.
With the patch applied, you can use -DUSE_KMALLOC to use kmalloc for allocation
instead.

====
# time stap -DUSE_KMALLOC --disable-cache
../src/testsuite/systemtap.maps/ix_clear.stp
foo[1] = 1 1
ERROR: empty aggregate near operator '@min' at
../src/testsuite/systemtap.maps/ix_clear.stp:14:45
WARNING: Number of errors: 1, skipped probes: 0
WARNING: /usr/local/bin/staprun exited with status: 1
Pass 5: run failed.  [man error::pass5]

real    0m20.055s
user    0m16.192s
sys    0m2.734s
====

So, using kmalloc(), it takes around 20 seconds to run that test (which is
supposed to fail).

Using vmalloc():

====
time stap --disable-cache ../src/testsuite/systemtap.maps/ix_clear.stp
foo[1] = 1 1
ERROR: empty aggregate near operator '@min' at
../src/testsuite/systemtap.maps/ix_clear.stp:14:45
WARNING: Number of errors: 1, skipped probes: 0
WARNING: /usr/local/bin/staprun exited with status: 1
Pass 5: run failed.  [man error::pass5]

real    11m34.415s
user    0m17.374s
sys    11m15.986s
====

With vmalloc, it takes over 11 minutes.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15805] maps should not be allocated with k*alloc

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15805

--- Comment #4 from David Smith <dsmith at redhat dot com> ---
Here's some additional information. After some more debugging, I've found out
the systemtap.maps/ix_clear.stp testcase ends up making one call to
_stp_pmap_new() with the following parameters: max_entries: 2048, wrap: 0,
node_size 72. This ends up with 67618 calls to  _stp_map_vzalloc().

That's right, over 67000 calls to _stp_map_vzalloc() from 1 call to
_stp_pmap_new().

Looks like if we switch to vmalloc, we need to figure out how to allocate in
bigger chunks.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15805] maps should not be allocated with k*alloc

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15805

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #7135|0                           |1
        is obsolete|                            |

--- Comment #5 from David Smith <dsmith at redhat dot com> ---
Created attachment 7143
  --> http://sourceware.org/bugzilla/attachment.cgi?id=7143&action=edit
v2 patch

Here's a new version of the patch that takes allocations from over 67000 to 67.
With this new patch, here are some timed runs:

====
# time stap -DUSE_KMALLOC --disable-cache
../src/testsuite/systemtap.maps/ix_clear.stp 
foo[1] = 1 1
ERROR: empty aggregate near operator '@min' at
../src/testsuite/systemtap.maps/ix_clear.stp:14:45
WARNING: Number of errors: 1, skipped probes: 0
WARNING: /usr/local/bin/staprun exited with status: 1
Pass 5: run failed.  [man error::pass5]

real    0m20.810s
user    0m17.327s
sys    0m2.425s
# time stap --disable-cache ../src/testsuite/systemtap.maps/ix_clear.stp 
foo[1] = 1 1
ERROR: empty aggregate near operator '@min' at
../src/testsuite/systemtap.maps/ix_clear.stp:14:45
WARNING: Number of errors: 1, skipped probes: 0
WARNING: /usr/local/bin/staprun exited with status: 1
Pass 5: run failed.  [man error::pass5]

real    0m20.704s
user    0m17.352s
sys    0m2.306s
====

With this version of the patch, the new kmalloc and vmalloc versions of the
code times are very similar. With this patch, all of the systemtap.maps/*.exp
tests pass. I haven't done a full testsuite run.

Note that allocations could be lowered further by using ideas from the dyninst
runtime, which only makes 1 allocation per map.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug runtime/15805] maps should not be allocated with k*alloc

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15805

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #6 from David Smith <dsmith at redhat dot com> ---
Fixed in commit 3f873e5.

-- 
You are receiving this mail because:
You are the assignee for the bug.