[Bug kprobes/13819] New: A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[Bug kprobes/13819] New: A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[kevincolbert33 at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

             Bug #: 13819
           Summary: A Call Trace is generated when stap is run if the
                    kernel option "Debug access to per_cpu maps" is
                    enabled
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: kprobes
        AssignedTo: systemtap@sourceware.org
        ReportedBy: kevincolbert33@gmail.com
    Classification: Unclassified


Created attachment 6266
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6266
stap-report

SystemTap version 1.7 - If the kernel option "Debug access to per_cpu maps" is
enabled a Call Trace is generated when stap is run. The script parameters that
stap executes doesn't seem to make a difference, every script that I tried
would cause the Call Trace to be generated.  A simple one liner like "stap -v
-e 'probe kernel.function("vfs_read") {printf("%s\n", $$parms); exit(); }'"
will generate the Call Trace. If the "Debug access to per_cpu maps" kernel
option is disabled the Call Trace will not be generated.  Please see the
attached stap-report log file for all of the details.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[kevincolbert33 at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

Kevin Colbert <kevincolbert33 at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|---                         |weeks

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> 2012-03-08 00:24:02 UTC ---
Kevin, can you rerun the stap script with -DDEBUG_TRANS, and attach
the resulting dmesg lines?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[kevincolbert33 at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

--- Comment #2 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 00:35:15 UTC ---
Mar  7 18:28:00 localhost kernel: [ 5089.611692] _stp_transport_init:310
transport_init
Mar  7 18:28:00 localhost kernel: [ 5089.611702] _stp_transport_fs_init:545
entry
Mar  7 18:28:00 localhost kernel: [ 5089.613174]
_stp_transport_data_fs_init:357 returning 0...
Mar  7 18:28:00 localhost kernel: [ 5089.613195] _stp_transport_fs_init:592
returning 0
Mar  7 18:28:00 localhost kernel: [ 5089.613674] _stp_ctl_send_notify:468
_stp_ctl_send_notify: type=4 len=0
Mar  7 18:28:00 localhost kernel: [ 5089.613679] _stp_ctl_write_dbug:170
sending STP_TRANSPORT
Mar  7 18:28:00 localhost kernel: [ 5089.613776] _stp_transport_init:387
returning 0...
Mar  7 18:28:00 localhost kernel: [ 5089.614430] _stp_attach:245 attach
Mar  7 18:28:00 localhost kernel: [ 5089.719808] _stp_detach:227 detach
Mar  7 18:28:00 localhost kernel: [ 5089.722608] _stp_attach:245 attach
Mar  7 18:28:00 localhost kernel: [ 5089.723362] _stp_handle_start:107
stp_handle_start
Mar  7 18:28:00 localhost kernel: [ 5089.724317] _stp_ctl_send_notify:468
_stp_ctl_send_notify: type=0 len=8
Mar  7 18:28:00 localhost kernel: [ 5089.724325] _stp_ctl_write_dbug:155
sending STP_START
Mar  7 18:28:00 localhost kernel: [ 5089.724349] stp_print_flush:27 len = 79
Mar  7 18:28:00 localhost kernel: [ 5089.724356] stp_print_flush:112 calling
_stp_data_write...
Mar  7 18:28:00 localhost kernel: [ 5089.742299] _stp_request_exit:214 ctl_send
STP_REQUEST_EXIT
Mar  7 18:28:00 localhost kernel: [ 5089.742305] _stp_ctl_send_notify:468
_stp_ctl_send_notify: type=13 len=0
Mar  7 18:28:00 localhost kernel: [ 5089.742309] _stp_ctl_write_dbug:191
sending STP_REQUEST_EXIT
Mar  7 18:28:00 localhost kernel: [ 5089.742375] _stp_request_exit:218 done
with ctl_send STP_REQUEST_EXIT
Mar  7 18:28:00 localhost kernel: [ 5089.742408] _stp_cleanup_and_exit:179
cleanup_and_exit (1)
Mar  7 18:28:00 localhost kernel: [ 5089.742412] _stp_cleanup_and_exit:183
calling systemtap_module_exit
Mar  7 18:28:00 localhost kernel: [ 5089.759333] ------------[ cut here
]------------
Mar  7 18:28:00 localhost kernel: [ 5089.759344] WARNING: at
include/linux/cpumask.h:107 _stp_cleanup_and_exit+0x72f/0x7b0
[stap_a97c8ec2c8cf617f051c014435d8912a_4811]() (Tainted: G        W 
----------------  )
Mar  7 18:28:00 localhost kernel: [ 5089.759349] Hardware name: ProLiant DL380
G7
Mar  7 18:28:00 localhost kernel: [ 5089.759352] Modules linked in:
stap_a97c8ec2c8cf617f051c014435d8912a_4811(U) ip6table_filter ip6_tables
ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4
nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle
iptable_filter ip_tables bridge autofs4 sunrpc bnx2fc cnic uio fcoe libfcoe
8021q garp libfc stp scsi_transport_fc llc scsi_tgt pcc_cpufreq ipv6 dm_mirror
dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun kvm_intel kvm uinput
sg power_meter microcode iTCO_wdt iTCO_vendor_support hpilo hpwdt bnx2
i7core_edac edac_core shpchp ext4 mbcache jbd2 cciss hpsa radeon ttm
drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded:
stap_a97c8ec2c8cf617f051c014435d8912a_4586]
Mar  7 18:28:00 localhost kernel: [ 5089.759430] Pid: 4811, comm: stapio
Tainted: G        W  ----------------   2.6.32 #1
Mar  7 18:28:00 localhost kernel: [ 5089.759434] Call Trace:
Mar  7 18:28:00 localhost kernel: [ 5089.759442]  [<ffffffff810a1027>] ?
warn_slowpath_common+0xc7/0x130
Mar  7 18:28:00 localhost kernel: [ 5089.759447]  [<ffffffff810a10ba>] ?
warn_slowpath_null+0x2a/0x40
Mar  7 18:28:00 localhost kernel: [ 5089.759455]  [<ffffffffa0791e0f>] ?
_stp_cleanup_and_exit+0x72f/0x7b0 [stap_a97c8ec2c8cf617f051c014435d8912a_4811]
Mar  7 18:28:00 localhost kernel: [ 5089.759463]  [<ffffffffa0792e62>] ?
_stp_ctl_write_cmd+0x2b2/0xef0 [stap_a97c8ec2c8cf617f051c014435d8912a_4811]
Mar  7 18:28:00 localhost kernel: [ 5089.759470]  [<ffffffff81259716>] ?
fget_light+0x76/0x150
Mar  7 18:28:00 localhost kernel: [ 5089.759476]  [<ffffffff81273a65>] ?
sys_fcntl+0x745/0x8a0
Mar  7 18:28:00 localhost kernel: [ 5089.759483]  [<ffffffff8133bdde>] ?
security_file_permission+0x1e/0x30
Mar  7 18:28:00 localhost kernel: [ 5089.759488]  [<ffffffff81257a89>] ?
vfs_write+0xb9/0x250
Mar  7 18:28:00 localhost kernel: [ 5089.759493]  [<ffffffff81259716>] ?
fget_light+0x76/0x150
Mar  7 18:28:00 localhost kernel: [ 5089.759499]  [<ffffffff811551b3>] ?
audit_syscall_entry+0x363/0x3a0
Mar  7 18:28:00 localhost kernel: [ 5089.759505]  [<ffffffff81258639>] ?
sys_write+0x69/0xd0
Mar  7 18:28:00 localhost kernel: [ 5089.759510]  [<ffffffff8100ff72>] ?
system_call_fastpath+0x16/0x1b
Mar  7 18:28:00 localhost kernel: [ 5089.759514] ---[ end trace
f8dbfc3753fd3ed3 ]---
Mar  7 18:28:00 localhost kernel: [ 5090.014282] _stp_cleanup_and_exit:186 done
with systemtap_module_exit
Mar  7 18:28:00 localhost kernel: [ 5090.014287] _stp_cleanup_and_exit:193 ***
calling _stp_transport_data_fs_stop ***
Mar  7 18:28:00 localhost kernel: [ 5090.014293]
_stp_transport_data_fs_stop:263 flushing...
Mar  7 18:28:00 localhost kernel: [ 5090.014298] _stp_cleanup_and_exit:196
ctl_send STP_EXIT
Mar  7 18:28:00 localhost kernel: [ 5090.014302] _stp_ctl_send_notify:468
_stp_ctl_send_notify: type=1 len=0
Mar  7 18:28:00 localhost kernel: [ 5090.014306] _stp_ctl_write_dbug:158
sending STP_EXIT
Mar  7 18:28:00 localhost kernel: [ 5090.014311] _stp_cleanup_and_exit:204 done
with ctl_send STP_EXIT
Mar  7 18:28:00 localhost kernel: [ 5090.014624] _stp_detach:227 detach
Mar  7 18:28:00 localhost kernel: [ 5090.023361] _stp_attach:245 attach
Mar  7 18:28:00 localhost kernel: [ 5090.023401] _stp_detach:227 detach
Mar  7 18:28:00 localhost kernel: [ 5090.023564] _stp_transport_close:294 4816:
************** transport_close *************
Mar  7 18:28:00 localhost kernel: [ 5090.023711] _stp_transport_fs_close:598
stp_transport_fs_close
Mar  7 18:28:00 localhost kernel: [ 5090.025134] _stp_transport_close:301 ----
CLOSED ----
Mar  7 18:28:01 localhost abrtd: Directory 'oops-2012-03-07-18:28:01-2159-0'
creation detected
Mar  7 18:28:01 localhost abrt-dump-oops: Reported 1 kernel oopses to Abrt
Mar  7 18:28:01 localhost abrtd: Can't open file
'/var/spool/abrt/oops-2012-03-07-18:28:01-2159-0/uid': No such file or
directory
Mar  7 18:28:15 localhost abrtd: Sending an email...
Mar  7 18:28:15 localhost abrtd: Email was sent to: root@localhost
Mar  7 18:28:15 localhost abrtd: Can't open file
'/var/spool/abrt/oops-2012-03-07-15:21:27-5527-0/uid': No such file or
directory
Mar  7 18:28:15 localhost abrtd: DUP_OF_DIR:
/var/spool/abrt/oops-2012-03-07-15:21:27-5527-0
Mar  7 18:28:15 localhost abrtd: Dump directory is a duplicate of
/var/spool/abrt/oops-2012-03-07-15:21:27-5527-0
Mar  7 18:28:15 localhost abrtd: Deleting dump directory
oops-2012-03-07-18:28:01-2159-0 (dup of oops-2012-03-07-15:21:27-5527-0),
sending dbus signal

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[kevincolbert33 at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

--- Comment #4 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 00:45:17 UTC ---
Created attachment 6268
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6268
attachment-17517-1.dat

--- Comment #5 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 00:45:17 UTC ---
Created attachment 6269
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6269
additional_data.txt

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[kevincolbert33 at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

--- Comment #3 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 00:45:15 UTC ---
I have attached a file that contains the dmesg output that was generated
using the -DDEBUG_TRANS switch.

Kevin
On Wed, Mar 7, 2012 at 6:24 PM, fche at redhat dot com <
sourceware-bugzilla@sourceware.org> wrote:

> http://sourceware.org/bugzilla/show_bug.cgi?id=13819
>
> Frank Ch. Eigler <fche at redhat dot com> changed:
>
>           What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>                 CC|                            |fche at redhat dot com
>
> --- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> 2012-03-08
> 00:24:02 UTC ---
> Kevin, can you rerun the stap script with -DDEBUG_TRANS, and attach
> the resulting dmesg lines?
>
> --
> Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You reported the bug.
>

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

--- Comment #6 from Frank Ch. Eigler <fche at redhat dot com> 2012-03-08 03:00:03 UTC ---
Created attachment 6270
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6270
speculative translate.cxx patch

Just in case something's wacky about NR_CPUS on your machine,
can you try the following patch to translate.cxx?

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[kevincolbert33 at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

--- Comment #8 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 16:51:28 UTC ---
Created attachment 6272
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6272
attachment-11122-0.html

--- Comment #9 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 16:51:28 UTC ---
Created attachment 6273
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6273
attachment-11122-1.dat

--- Comment #10 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 16:51:28 UTC ---
Created attachment 6274
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6274
stap_compilation_output.txt

--- Comment #11 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 16:51:28 UTC ---
Created attachment 6275
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6275
translate.cxx_snippet.txt

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[kevincolbert33 at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

--- Comment #7 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 16:51:25 UTC ---
Hi Frank,

I modified the translate.cxx file per your instructions, however after
recompiling SystemTap 1.7 with the changes, stap would fail when it tried
to compile the script.  I have attached two files, one shows the output
from stap and the other is a code snippet from my translate.cxx file.

Kevin

On Wed, Mar 7, 2012 at 9:00 PM, fche at redhat dot com <
sourceware-bugzilla@sourceware.org> wrote:

> http://sourceware.org/bugzilla/show_bug.cgi?id=13819
>
> --- Comment #6 from Frank Ch. Eigler <fche at redhat dot com> 2012-03-08
> 03:00:03 UTC ---
> Created attachment 6270
>  --> http://sourceware.org/bugzilla/attachment.cgi?id=6270
> speculative translate.cxx patch
>
> Just in case something's wacky about NR_CPUS on your machine,
> can you try the following patch to translate.cxx?
>
> --
> Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You reported the bug.
>

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #6270|0                           |1
        is obsolete|                            |

--- Comment #12 from Frank Ch. Eigler <fche at redhat dot com> 2012-03-08 20:23:48 UTC ---
Created attachment 6276
  --> http://sourceware.org/bugzilla/attachment.cgi?id=6276
corrected typo in speculative patch

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[kevincolbert33 at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

--- Comment #13 from Kevin Colbert <kevincolbert33 at gmail dot com> 2012-03-08 21:53:54 UTC ---
Hello Frank,

This patch fixed the issue.  The Call Trace does not occur any longer when
I run my SystemTap scripts.

Thank you for the fast response!!!

Kevin

On Thu, Mar 8, 2012 at 2:23 PM, fche at redhat dot com <
sourceware-bugzilla@sourceware.org> wrote:

> http://sourceware.org/bugzilla/show_bug.cgi?id=13819
>
> Frank Ch. Eigler <fche at redhat dot com> changed:
>
>           What    |Removed                     |Added
>
> ----------------------------------------------------------------------------
>   Attachment #6270|0                           |1
>        is obsolete|                            |
>
> --- Comment #12 from Frank Ch. Eigler <fche at redhat dot com> 2012-03-08
> 20:23:48 UTC ---
> Created attachment 6276
>  --> http://sourceware.org/bugzilla/attachment.cgi?id=6276
> corrected typo in speculative patch
>
> --
> Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You reported the bug.
>

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug kprobes/13819] A Call Trace is generated when stap is run if the kernel option "Debug access to per_cpu maps" is enabled

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=13819

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #14 from Frank Ch. Eigler <fche at redhat dot com> 2012-03-08 23:23:22 UTC ---
commit ebbfed0

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.