[Bug uprobes/17867] New: Kernel panic while running a simple script doing userland probing only

[Bug uprobes/17867] New: Kernel panic while running a simple script doing userland probing only

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17867

            Bug ID: 17867
           Summary: Kernel panic while running a simple script doing
                    userland probing only
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: uprobes
          Assignee: systemtap at sourceware dot org
          Reporter: agentzh at gmail dot com

Created attachment 8076
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8076&action=edit
Kernel panic logs

Hi, guys!

I was running the ngx-count-conns.stp script with systemtap 2.5 + kernel
3.10.27 on one of our metal boxes for several times. For many times it works as
expected, but for at least one time it triggers kernel panic (please see the
attached logs).

It's quite hard to reproduce. Not sure if it's a known issue that has already
been fixed in newer versions of systemtap and kernel.

We haven't seen any kernel panic with out extensive production use of
systemtap. So this is really for the first time and is quite scary ;) Any
pointers will be highly appreciated :)

More details:

    $ stap --version
    Systemtap translator/driver (version 2.5/0.159, non-git sources)
    Copyright (C) 2005-2014 Red Hat, Inc. and others
    This is free software; see the source for copying conditions.
    enabled features: TR1_UNORDERED_MAP NLS

    $ uname -a
    Linux 16m3 3.10.27 #1 SMP Fri Jan 17 12:03:42 PST 2014 x86_64 GNU/Linux

    $ gcc -v
    Using built-in specs.
    COLLECT_GCC=gcc
    COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
    Target: x86_64-linux-gnu
    Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.3-7'
--with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs
--enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-4.7 --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
--with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls
--with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug
--enable-gnu-unique-object --enable-plugin --with-system-zlib --enable-objc-gc
--enable-multiarch --with-arch-32=i586 --with-abi=m64
--with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
    Thread model: posix
    gcc version 4.7.3 (Debian 4.7.3-7)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/17867] Kernel panic while running a simple script doing userland probing only

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17867

--- Comment #1 from agentzh <agentzh at gmail dot com> ---
Created attachment 8077
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8077&action=edit
The systemtap script triggering the kernel panic

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/17867] Kernel panic while running a simple script doing userland probing only

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17867

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING
                 CC|                            |fche at redhat dot com

--- Comment #2 from Frank Ch. Eigler <fche at redhat dot com> ---
Many kernel uprobes improvements have been made since upstream 3.10,
so with that the newer the better.

I don't see any obvious systemtap implication in the kernel crash
though.  Can it be just a coincidence?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/17867] Kernel panic while running a simple script doing userland probing only

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17867

--- Comment #3 from agentzh <agentzh at gmail dot com> ---
Thanks for your reply, Frank!

Yes, we're going to test out the 3.14 kernel with backported uprobes patches
for fixing the relative CALL instruction emulation bug and the tmpfs
regression. Do you think if we need to backport even more patches from 3.15+ or
just upgrade to even a newer kernel like 3.18?

I'm not really sure if it's a coincidence. We haven't had any kernel panic in
our production environment for long. This panic just happened right after I
started my systemtap tool :)

Do you think if it'll be helpful to enable the kernel crash dump for offline
analysis if the issue happens again?

Thanks a lot for your time!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/17867] Kernel panic while running a simple script doing userland probing only

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17867

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |WORKSFORME

--- Comment #4 from Frank Ch. Eigler <fche at redhat dot com> ---
more recent kernels (>> 3.15) seem to have fairly well-behaved uprobes

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug uprobes/17867] Kernel panic while running a simple script doing userland probing only

[agentzh at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=17867

--- Comment #5 from agentzh <agentzh at gmail dot com> ---
Thanks for the info, Frank!

-- 
You are receiving this mail because:
You are the assignee for the bug.