public inbox for systemtap@sourceware.org
 help / color / mirror / Atom feed
* sendmail function probe registration error with systemtap
@ 2014-01-09 23:12 Xiaokui Shu
  2014-01-10 18:46 ` Frank Ch. Eigler
  0 siblings, 1 reply; 6+ messages in thread
From: Xiaokui Shu @ 2014-01-09 23:12 UTC (permalink / raw)
  To: systemtap

Hello everyone,

I am using systemtap to trace function calls of a running process. It
works well with small programs, e.g., grep. However, I run into an
issue with "sendmail". When I start systemtap with a simple probe
(shown below), I get two warnings in Pass 5. While running systemtap,
I can send emails to the monitored host, but my mailbox on the host
does not receive them. Only after I shutdown systemtap, the emails are
coming to /var/mail/xxx. When running systemtap, after sending several
emails, sendmail seems to be blocked, and all TCP connections to 25
port is rejected. After I shutdown systemtap, sendmail works again,
and the port is open.

Can anyone help me to figure out what is the problem brought by the tracing?

Thank you very much!

######## systemtap running log ########
Pass 1: parsed user script and 102 library script(s) using
32296virt/18852res/2588shr/16968data kb, in 280usr/10sys/297real ms.
Pass 2: analyzed script: 947 probe(s), 12 function(s), 3 embed(s), 0
global(s) using 41316virt/27460res/4520shr/23536data kb, in
210usr/160sys/369real ms.
Pass 3: using cached
/root/.systemtap/cache/56/stap_560093aa0c2089c2c95856502772a700_258364.c
Pass 4: using cached
/root/.systemtap/cache/56/stap_560093aa0c2089c2c95856502772a700_258364.ko
Pass 5: starting run.
WARNING: probe process("/usr/sbin/sendmail.sendmail").function("mime_getchar_crlf.part.1")
inode-offset 0005e620 registration error (rc -524)
WARNING: probe process("/usr/sbin/sendmail.sendmail").function("frame_dummy")
inode-offset 0001cc80 registration error (rc -524)

######## trace.stp ########
probe process(@1).function("*")
{
    printf ("%s:0x%x\n", probefunc (), ustack (1))
}

Best,
Xiaokui

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: sendmail function probe registration error with systemtap
  2014-01-09 23:12 sendmail function probe registration error with systemtap Xiaokui Shu
@ 2014-01-10 18:46 ` Frank Ch. Eigler
       [not found]   ` <CAMU3N_KaOPbsYQS1io=_v-g4PzzQ4mAbk0GAOveN2tf-dLmaCg@mail.gmail.com>
  0 siblings, 1 reply; 6+ messages in thread
From: Frank Ch. Eigler @ 2014-01-10 18:46 UTC (permalink / raw)
  To: Xiaokui Shu; +Cc: systemtap

Xiaokui Shu <subbyte@gmail.com> writes:

> [...] I start systemtap with a simple probe
> (shown below), I get two warnings in Pass 5. While running systemtap,
> [...]
> WARNING: probe process("/usr/sbin/sendmail.sendmail").function("mime_getchar_crlf.part.1")
> inode-offset 0005e620 registration error (rc -524)
> WARNING: probe process("/usr/sbin/sendmail.sendmail").function("frame_dummy")
> inode-offset 0001cc80 registration error (rc -524)

That's -ENOTSUPP; maybe there is some sort of instruction in there the
kernel doesn't like.

> I can send emails to the monitored host, but my mailbox on the host
> does not receive them. [...]  Can anyone help me to figure out what
> is the problem brought by the tracing?

This sounds should not happen.

> probe process(@1).function("*")
> {
>     printf ("%s:0x%x\n", probefunc (), ustack (1))
> }

You might try rerunning '# stap -DSTP_ALIBI foo.stp' to eliminate most
of the systemtap probe handler logic out of the script.  If the
userspace sendmail process still misbehaves, this helps incriminate
uprobes -- and vice versa.

Could you collect kernel / stap version information, or the whole
`stap-report` shebang?


- FChE

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Fwd: sendmail function probe registration error with systemtap
       [not found]   ` <CAMU3N_KaOPbsYQS1io=_v-g4PzzQ4mAbk0GAOveN2tf-dLmaCg@mail.gmail.com>
@ 2014-01-10 20:27     ` Xiaokui Shu
  2014-01-10 21:15     ` Frank Ch. Eigler
  1 sibling, 0 replies; 6+ messages in thread
From: Xiaokui Shu @ 2014-01-10 20:27 UTC (permalink / raw)
  To: systemtap

Thank you for your reply!

I find nothing changed with -DSTP_ALIBI. It does not stop sendmail
from not receiving emails. Do you suggest this is a uprobe problem
instead of systemtap?

The command I run stap is:
#stap -DSTP_ALIBI -o sendmail.trace -v trace.stp sendmail

Here's my stap-report information.

== id ==
uid=0(root) gid=0(root) groups=0(root)
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
== stap -V ==
Systemtap translator/driver (version 2.4/0.156, rpm 2.4-1.fc19)
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR
TR1_UNORDERED_MAP NLS DYNINST JAVA LIBVIRT LIBXML2
== which stap ==
/usr/bin/stap
== cat /root/.systemtap/rc ==
cat: /root/.systemtap/rc: No such file or directory
== which stap-server ==
which: no stap-server in
(/usr/lib/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/subx/.local/bin:/home/subx/bin:/bin:/sbin:/usr/bin:/usr/sbin)
== locate --regex '/stap(run|io|dyn)?$' | xargs ls -ald ==
-rwxr-xr-x. 1 root root    2724024 Nov  6 13:19 /usr/bin/stap
-rwxr-xr-x. 1 root root     173116 Nov  6 13:19 /usr/bin/stapdyn
---s--x---. 1 root stapusr  173128 Nov  6 13:19 /usr/bin/staprun
-rwxr-xr-x. 1 root root      56712 Nov  6 13:19 /usr/libexec/systemtap/stapio
== printenv | egrep
'^PATH=|^LD_LIBRARY_PATH=|^SYSTEMTAP_.*=|^XDG_DATA.*=|^NSS.*=' ==
PATH=/usr/lib/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/subx/.local/bin:/home/subx/bin:/bin:/sbin:/usr/bin:/usr/sbin
== stap -vv --poison-cache -p4 -e 'probe begin {exit()}' ==
Systemtap translator/driver (version 2.4/0.156, rpm 2.4-1.fc19)
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR
TR1_UNORDERED_MAP NLS DYNINST JAVA LIBVIRT LIBXML2
Created temporary directory "/tmp/stapu2dKKi"
Session arch: i386 release: 3.12.6-200.fc19.i686.PAE
Searched for library macro files:
"/usr/share/systemtap/tapset/linux/*.stpm", found: 2, processed: 2
Searched for library macro files:
"/usr/share/systemtap/tapset/*.stpm", found: 2, processed: 2
Searched: "/usr/share/systemtap/tapset/linux/i386/*.stp", found: 3, processed: 3
Searched: "/usr/share/systemtap/tapset/linux/*.stp", found: 67, processed: 67
Searched: "/usr/share/systemtap/tapset/i386/*.stp", found: 4, processed: 4
Searched: "/usr/share/systemtap/tapset/*.stp", found: 24, processed: 24
Pass 1: parsed user script and 102 library script(s) using
32140virt/18788res/2596shr/16812data kb, in 280usr/10sys/296real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0
global(s) using 32404virt/19316res/2680shr/17076data kb, in
10usr/0sys/8real ms.
function recursion-analysis: max-nesting 0  non-recursive
Pass 3: translated to C into
"/tmp/stapu2dKKi/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869_src.c"
using 32404virt/19656res/2984shr/17076data kb, in 0usr/0sys/0real ms.
Running env -uARCH -uKBUILD_EXTMOD -uCROSS_COMPILE -uKBUILD_IMAGE
-uKCONFIG_CONFIG -uINSTALL_PATH
PATH=/usr/bin:/bin:/usr/lib/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/subx/.local/bin:/home/subx/bin:/bin:/sbin:/usr/bin:/usr/sbin
make -C /lib/modules/3.12.6-200.fc19.i686.PAE/build M=/tmp/stapu2dKKi
modules ARCH=i386 CONFIG_DEBUG_INFO= --no-print-directory -j9
  CC [M]  /tmp/stapu2dKKi/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869_src.o
  LD [M]  /tmp/stapu2dKKi/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /tmp/stapu2dKKi/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869.mod.o
  LD [M]  /tmp/stapu2dKKi/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869.ko
Spawn waitpid result (0x0): 0
/root/.systemtap/cache/68/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869.ko
Pass 4: compiled C into "stap_68805bd3fd056ab1b2b2aa38a5a6865d_869.ko"
in 8370usr/1350sys/9925real ms.
Cleaning cache, interval reached 339 s > 300 s.
Copying /tmp/stapu2dKKi/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869.ko
to /root/.systemtap/cache/68/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869.ko
Copying /tmp/stapu2dKKi/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869_src.c
to /root/.systemtap/cache/68/stap_68805bd3fd056ab1b2b2aa38a5a6865d_869.c
Copying /tmp/stapu2dKKi/stapconf_4dbe13844bb0f5e88692aa83d9fe7b63_685.h
to /root/.systemtap/cache/4d/stapconf_4dbe13844bb0f5e88692aa83d9fe7b63_685.h
Running rm -rf /tmp/stapu2dKKi
Spawn waitpid result (0x0): 0
Removed temporary directory "/tmp/stapu2dKKi"
== gcc -v ==
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/i686-redhat-linux/4.8.2/lto-wrapper
Target: i686-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap
--enable-shared --enable-threads=posix --enable-checking=release
--with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-gnu-unique-object
--enable-linker-build-id --with-linker-hash-style=gnu
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto
--enable-plugin --enable-initfini-array --enable-java-awt=gtk
--disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
--enable-libgcj-multifile --enable-java-maintainer-mode
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar
--disable-libjava-multilib
--with-isl=/builddir/build/BUILD/gcc-4.8.2-20131212/obj-i686-redhat-linux/isl-install
--with-cloog=/builddir/build/BUILD/gcc-4.8.2-20131212/obj-i686-redhat-linux/cloog-install
--with-tune=generic --with-arch=i686 --build=i686-redhat-linux
Thread model: posix
gcc version 4.8.2 20131212 (Red Hat 4.8.2-7) (GCC)
== uname -a ==
Linux xxx 3.12.6-200.fc19.i686.PAE #1 SMP Mon Dec 23 16:58:40 UTC 2013
i686 i686 i386 GNU/Linux
== dmesg | fgrep 'Linux version' ==
[    0.000000] Linux version 3.12.6-200.fc19.i686.PAE
(mockbuild@bkernel02) (gcc version 4.8.2 20131212 (Red Hat 4.8.2-7)
(GCC) ) #1 SMP Mon Dec 23 16:58:40 UTC 2013
== dmesg | egrep 'stap|systemtap' | tail -n 10 ==
[88314.093413] stap_560093aa0c2089c2c95856502772a700_2_6438:
systemtap: 2.4/0.156, base: fb312000, memory:
594data/60text/46ctx/8202net/66alloc kb, probes: 947
[93179.056866] stap_560093aa0c2089c2c95856502772a700_2_6770:
systemtap: 2.4/0.156, base: fa9f7000, memory:
594data/60text/46ctx/8202net/66alloc kb, probes: 947
[93331.071995] stap_560093aa0c2089c2c95856502772a700_2_6811:
systemtap: 2.4/0.156, base: fb312000, memory:
594data/60text/46ctx/8202net/66alloc kb, probes: 947
[175306.234692] stap_560093aa0c2089c2c95856502772a700_2_8906:
systemtap: 2.4/0.156, base: fb312000, memory:
594data/60text/46ctx/8202net/66alloc kb, probes: 947
[181559.185750] stap_560093aa0c2089c2c95856502772a700_2_9048:
systemtap: 2.4/0.156, base: fb3b7000, memory:
594data/60text/46ctx/8202net/66alloc kb, probes: 947
[181767.879292] stap_560093aa0c2089c2c95856502772a700_2_9063:
systemtap: 2.4/0.156, base: fb312000, memory:
594data/60text/46ctx/8202net/66alloc kb, probes: 947
[257643.097467] stap_5e3d2cbe1be0f4b0a2ae52b1a5b2552f__10702:
systemtap: 2.4/0.156, base: fb42e000, memory:
594data/60text/47ctx/8202net/66alloc kb, probes: 947
[257702.217859] stap_5e00cd108e5512e0ce632b330f0f468e__11143:
systemtap: 2.4/0.156, base: fb4d3000, memory:
694data/56text/47ctx/2058net/66alloc kb, probes: 947
[257860.763992] stap_5e00cd108e5512e0ce632b330f0f468e__11222:
systemtap: 2.4/0.156, base: fb3a1000, memory:
694data/56text/47ctx/2058net/66alloc kb, probes: 947
[257887.565522] stap_5e00cd108e5512e0ce632b330f0f468e__11253:
systemtap: 2.4/0.156, base: fb45e000, memory:
694data/56text/47ctx/2058net/66alloc kb, probes: 947

== cat /etc/hosts ==
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
== cat /var/log/stap-server/log ==
cat: /var/log/stap-server/log: No such file or directory
== cat /proc/cpuinfo | egrep 'processor|vendor_id|model name' ==
processor : 0
vendor_id : GenuineIntel
model name : Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
processor : 1
vendor_id : GenuineIntel
model name : Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
processor : 2
vendor_id : GenuineIntel
model name : Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
processor : 3
vendor_id : GenuineIntel
model name : Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
processor : 4
vendor_id : GenuineIntel
model name : Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
processor : 5
vendor_id : GenuineIntel
model name : Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
processor : 6
vendor_id : GenuineIntel
model name : Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
processor : 7
vendor_id : GenuineIntel
model name : Intel(R) Xeon(R) CPU           E5320  @ 1.86GHz
== rpm -qa --qf '%{name}-%{version}
%{release}.%{arch}\t%{buildtime:date}@%{buildhost}\n' | egrep
'systemtap|elfutils|kernel|gcc|dyninst|avahi|nss|nspr|dejagnu|libdw' |
sort ==
avahi-0.6.31 11.fc19.i686 Sat 02 Feb 2013 09:25:56 AM
EST@buildvm-02.phx2.fedoraproject.org
avahi-autoipd-0.6.31 11.fc19.i686 Sat 02 Feb 2013 09:25:56 AM
EST@buildvm-02.phx2.fedoraproject.org
avahi-glib-0.6.31 11.fc19.i686 Sat 02 Feb 2013 09:25:56 AM
EST@buildvm-02.phx2.fedoraproject.org
avahi-gobject-0.6.31 11.fc19.i686 Sat 02 Feb 2013 09:25:56 AM
EST@buildvm-02.phx2.fedoraproject.org
avahi-libs-0.6.31 11.fc19.i686 Sat 02 Feb 2013 09:25:56 AM
EST@buildvm-02.phx2.fedoraproject.org
dyninst-8.1.2 1.fc19.i686 Tue 18 Jun 2013 07:54:26 PM
EDT@buildvm-15.phx2.fedoraproject.org
dyninst-devel-8.1.2 1.fc19.i686 Tue 18 Jun 2013 07:54:26 PM
EDT@buildvm-15.phx2.fedoraproject.org
elfutils-0.156 5.fc19.i686 Fri 06 Sep 2013 01:22:03 PM
EDT@buildvm-22.phx2.fedoraproject.org
elfutils-libelf-0.156 5.fc19.i686 Fri 06 Sep 2013 01:22:03 PM
EDT@buildvm-22.phx2.fedoraproject.org
elfutils-libs-0.156 5.fc19.i686 Fri 06 Sep 2013 01:22:03 PM
EDT@buildvm-22.phx2.fedoraproject.org
fedora-release-19 5.noarch Thu 14 Nov 2013 12:20:30 AM EST@bkernel02
gcc-4.8.2 7.fc19.i686 Thu 12 Dec 2013 05:37:18 PM
EST@buildvm-18.phx2.fedoraproject.org
gcc-base-debuginfo-4.8.2 7.fc19.i686 Thu 12 Dec 2013 05:37:18 PM
EST@buildvm-18.phx2.fedoraproject.org
gcc-c++-4.8.2 7.fc19.i686 Thu 12 Dec 2013 05:37:18 PM
EST@buildvm-18.phx2.fedoraproject.org
gcc-debuginfo-4.8.2 7.fc19.i686 Thu 12 Dec 2013 05:37:18 PM
EST@buildvm-18.phx2.fedoraproject.org
grub2-2.00 23.fc19.i686 Tue 02 Jul 2013 02:29:43 PM EDT@bkernel02
grub2-tools-2.00 23.fc19.i686 Tue 02 Jul 2013 02:29:43 PM EDT@bkernel02
kernel-headers-3.12.6 200.fc19.i686 Mon 23 Dec 2013 01:04:50 PM EST@bkernel02
kernel-PAE-3.11.10 200.fc19.i686 Mon 02 Dec 2013 04:42:34 PM
EST@bkernel01.phx2.fedoraproject.org
kernel-PAE-3.11.9 200.fc19.i686 Wed 20 Nov 2013 05:35:02 PM
EST@bkernel01.phx2.fedoraproject.org
kernel-PAE-3.12.6 200.fc19.i686 Mon 23 Dec 2013 01:04:50 PM EST@bkernel02
kernel-PAE-devel-3.11.10 200.fc19.i686 Mon 02 Dec 2013 04:42:34 PM
EST@bkernel01.phx2.fedoraproject.org
kernel-PAE-devel-3.11.9 200.fc19.i686 Wed 20 Nov 2013 05:35:02 PM
EST@bkernel01.phx2.fedoraproject.org
kernel-PAE-devel-3.12.6 200.fc19.i686 Mon 23 Dec 2013 01:04:50 PM EST@bkernel02
kernel-PAE-modules-extra-3.11.10 200.fc19.i686 Mon 02 Dec 2013
04:42:34 PM EST@bkernel01.phx2.fedoraproject.org
kernel-PAE-modules-extra-3.11.9 200.fc19.i686 Wed 20 Nov 2013 05:35:02
PM EST@bkernel01.phx2.fedoraproject.org
kernel-PAE-modules-extra-3.12.6 200.fc19.i686 Mon 23 Dec 2013 01:04:50
PM EST@bkernel02
libdwarf-20130207 1.fc19.i686 Fri 08 Feb 2013 09:54:07 AM
EST@buildvm-22.phx2.fedoraproject.org
libgcc-4.8.2 7.fc19.i686 Thu 12 Dec 2013 05:37:18 PM
EST@buildvm-18.phx2.fedoraproject.org
mingw32-gcc-4.8.2 1.fc19.i686 Sat 19 Oct 2013 05:28:13 PM
EDT@buildvm-07.phx2.fedoraproject.org
mingw32-gcc-c++-4.8.2 1.fc19.i686 Sat 19 Oct 2013 05:28:13 PM
EDT@buildvm-07.phx2.fedoraproject.org
nspr-4.10.2 1.fc19.i686 Mon 09 Dec 2013 11:36:58 AM
EST@buildvm-18.phx2.fedoraproject.org
nspr-debuginfo-4.10.2 1.fc19.i686 Mon 09 Dec 2013 11:36:58 AM
EST@buildvm-18.phx2.fedoraproject.org
nss-3.15.3.1 1.fc19.i686 Wed 18 Dec 2013 12:30:27 PM
EST@buildvm-17.phx2.fedoraproject.org
nss-softokn-3.15.3 1.fc19.i686 Mon 09 Dec 2013 02:25:13 PM
EST@buildvm-02.phx2.fedoraproject.org
nss-softokn-freebl-3.15.3 1.fc19.i686 Mon 09 Dec 2013 02:25:13 PM
EST@buildvm-02.phx2.fedoraproject.org
nss-sysinit-3.15.3.1 1.fc19.i686 Wed 18 Dec 2013 12:30:27 PM
EST@buildvm-17.phx2.fedoraproject.org
nss-tools-3.15.3.1 1.fc19.i686 Wed 18 Dec 2013 12:30:27 PM
EST@buildvm-17.phx2.fedoraproject.org
nss-util-3.15.3 1.fc19.i686 Mon 09 Dec 2013 01:27:00 PM
EST@buildvm-23.phx2.fedoraproject.org
openssh-6.2p2 7.fc19.i686 Wed 11 Dec 2013 09:52:52 AM
EST@buildvm-24.phx2.fedoraproject.org
openssh-clients-6.2p2 7.fc19.i686 Wed 11 Dec 2013 09:52:52 AM
EST@buildvm-24.phx2.fedoraproject.org
openssh-server-6.2p2 7.fc19.i686 Wed 11 Dec 2013 09:52:52 AM
EST@buildvm-24.phx2.fedoraproject.org
openssl-1.0.1e 36.fc19.i686 Fri 20 Dec 2013 09:03:17 AM
EST@buildvm-08.phx2.fedoraproject.org
openssl-devel-1.0.1e 36.fc19.i686 Fri 20 Dec 2013 09:03:17 AM
EST@buildvm-08.phx2.fedoraproject.org
openssl-libs-1.0.1e 36.fc19.i686 Fri 20 Dec 2013 09:03:17 AM
EST@buildvm-08.phx2.fedoraproject.org
python-nss-0.14.0 1.fc19.i686 Mon 13 May 2013 01:24:57 PM
EDT@buildvm-06.phx2.fedoraproject.org
systemtap-2.4 1.fc19.i686 Wed 06 Nov 2013 01:20:01 PM
EST@buildvm-09.phx2.fedoraproject.org
systemtap-client-2.4 1.fc19.i686 Wed 06 Nov 2013 01:20:01 PM
EST@buildvm-09.phx2.fedoraproject.org
systemtap-devel-2.4 1.fc19.i686 Wed 06 Nov 2013 01:20:01 PM
EST@buildvm-09.phx2.fedoraproject.org
systemtap-runtime-2.4 1.fc19.i686 Wed 06 Nov 2013 01:20:01 PM
EST@buildvm-09.phx2.fedoraproject.org
systemtap-sdt-devel-2.4 1.fc19.i686 Wed 06 Nov 2013 01:20:01 PM
EST@buildvm-09.phx2.fedoraproject.org
== egrep 'PROBE|TRACE|MARKER|KALLSYM|_DEBUG_|MODULE|FENTRY|_SIG'
/lib/modules/3.12.6-200.fc19.i686.PAE/build/.config | grep -v not.set
| sort | fmt -w 80 ==
CONFIG_AIC79XX_DEBUG_MASK=0 CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_ARCH_CPU_PROBE_RELEASE=y CONFIG_ARCH_HAS_CPU_AUTOPROBE=y
CONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_BLK_DEV_IO_TRACE=y CONFIG_CB710_DEBUG_ASSUMPTIONS=y
CONFIG_CHECK_SIGNATURE=y CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_DEBUG_BOOT_PARAMS=y CONFIG_DEBUG_BUGVERBOSE=y CONFIG_DEBUG_DEVRES=y
CONFIG_DEBUG_FS=y CONFIG_DEBUG_HIGHMEM=y CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_KERNEL=y CONFIG_DEBUG_LIST=y CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_NX_TEST=m CONFIG_DEBUG_RODATA_TEST=y
CONFIG_DEBUG_RODATA=y CONFIG_DEBUG_SET_MODULE_RONX=y
CONFIG_DEBUG_SHIRQ=y CONFIG_DEBUG_STACKOVERFLOW=y CONFIG_DEBUG_VM=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y CONFIG_DYNAMIC_FTRACE=y
CONFIG_EFI_SIGNATURE_LIST_PARSER=y CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE_SYSCALLS=y CONFIG_FTRACE=y CONFIG_FUNCTION_TRACER=y
CONFIG_GENERIC_IRQ_PROBE=y CONFIG_GENERIC_TRACER=y CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DEBUG_KMEMLEAK=y CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y CONFIG_INFINIBAND_IPOIB_DEBUG_DATA=y
CONFIG_KALLSYMS_ALL=y CONFIG_KALLSYMS=y CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_KPROBE_EVENT=y CONFIG_KPROBES_ON_FTRACE=y CONFIG_KPROBES=y
CONFIG_KRETPROBES=y CONFIG_MODULE_SIG_ALL=y CONFIG_MODULE_SIG_HASH="sha256"
CONFIG_MODULE_SIG_SHA256=y CONFIG_MODULE_SIG_UEFI=y CONFIG_MODULE_SIG=y
CONFIG_MODULES_USE_ELF_REL=y CONFIG_MODULES=y CONFIG_MODULE_UNLOAD=y
CONFIG_NCPFS_PACKET_SIGNING=y CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NET_SCTPPROBE=m CONFIG_NOP_TRACER=y CONFIG_NOUVEAU_DEBUG_DEFAULT=3
CONFIG_OLD_SIGACTION=y CONFIG_OLD_SIGSUSPEND3=y CONFIG_OPTPROBES=y
CONFIG_PM_TRACE_RTC=y CONFIG_PM_TRACE=y CONFIG_PROBE_EVENTS=y
CONFIG_SCHED_TRACER=y CONFIG_SCSI_IPR_TRACE=y CONFIG_SIGMATEL_FIR=m
CONFIG_SIGNALFD=y CONFIG_SND_HDA_CODEC_SIGMATEL=y CONFIG_STACK_TRACER=y
CONFIG_STACKTRACE_SUPPORT=y CONFIG_STACKTRACE=y CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_TRACE_CLOCK=y CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_TRACEPOINTS=y
CONFIG_TRACER_MAX_TRACE=y CONFIG_TRACER_SNAPSHOT=y CONFIG_UPROBE_EVENT=y
CONFIG_UPROBES=y CONFIG_USER_STACKTRACE_SUPPORT=y CONFIG_XEN_DEBUG_FS=y
== find /debugfs /proc /sys /dev /sys/kernel/debug -path '*kprobe*' -o
-path '*yama*' -type f 2>/dev/null | xargs grep -H . ==
/proc/sys/debug/kprobes-optimization:0
grep: /sys/kernel/debug/kprobes: Is a directory
/sys/kernel/debug/kprobes/enabled:1
grep: /sys/kernel/debug/kprobes: Is a directory
/sys/kernel/debug/kprobes/enabled:1
== lsmod ==
Module                  Size  Used by
binfmt_misc            13111  1
nfnetlink_queue        17785  0
nfnetlink_log          17489  0
ip6table_filter        12711  0
ip6_tables             17634  1 ip6table_filter
nfnetlink              13809  2 nfnetlink_log,nfnetlink_queue
bluetooth             317874  0
rfkill                 20934  1 bluetooth
udp_diag               12697  0
tcp_diag               12527  0
inet_diag              17838  2 tcp_diag,udp_diag
nf_conntrack_ipv4      14320  6
nf_defrag_ipv4         12601  1 nf_conntrack_ipv4
xt_recent              18105  2
xt_conntrack           12664  6
nf_conntrack           82832  2 xt_conntrack,nf_conntrack_ipv4
snd_hda_codec_idt      44642  1
snd_hda_intel          42658  1
snd_hda_codec         155840  2 snd_hda_codec_idt,snd_hda_intel
snd_hwdep              13232  1 snd_hda_codec
i2c_i801               17783  0
snd_seq                54679  0
snd_seq_device         13824  1 snd_seq
tg3                   151890  0
ptp                    18269  1 tg3
pps_core               18587  1 ptp
iTCO_wdt               13256  0
iTCO_vendor_support    13243  1 iTCO_wdt
lpc_ich                16864  0
mfd_core               13022  1 lpc_ich
coretemp               13195  0
i5000_edac             17213  0
edac_core              49939  2 i5000_edac
ppdev                  17363  0
i5k_amb                12962  0
serio_raw              13189  0
snd_pcm                82032  2 snd_hda_codec,snd_hda_intel
kvm_intel             132347  0
kvm                   368483  1 kvm_intel
dcdbas                 14371  0
parport_pc             27416  0
parport                39185  2 ppdev,parport_pc
microcode              18853  0
snd_page_alloc         13852  2 snd_pcm,snd_hda_intel
snd_timer              23750  2 snd_pcm,snd_seq
snd                    59132  10
snd_hwdep,snd_timer,snd_hda_codec_idt,snd_pcm,snd_seq,snd_hda_codec,snd_hda_intel,snd_seq_device
soundcore              14123  1 snd
nfsd                  243440  1
auth_rpcgss            48519  1 nfsd
nfs_acl                12653  1 nfsd
lockd                  78065  1 nfsd
sunrpc                233031  5 nfsd,auth_rpcgss,lockd,nfs_acl
nouveau               839813  2
video                  18595  1 nouveau
mxm_wmi                12761  1 nouveau
wmi                    18356  2 mxm_wmi,nouveau
i2c_algo_bit           13065  1 nouveau
drm_kms_helper         44756  1 nouveau
ttm                    64127  1 nouveau
firewire_ohci          35488  0
drm                   230658  4 ttm,drm_kms_helper,nouveau
ata_generic            12790  0
firewire_core          55963  1 firewire_ohci
pata_acpi              12886  0
i2c_core               28360  5 drm,i2c_i801,drm_kms_helper,i2c_algo_bit,nouveau
crc_itu_t              12549  1 firewire_core
== avahi-browse -r -t _stap._tcp ==
sh: avahi-browse: command not found
== ifconfig -a ==
em1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet xxx.xxx.xxx.xxx mask 255.255.252.0  broadcast xxx.xxx.xxx.xxx
        inet6 fe80::21a:a0ff:febe:62c0  prefixlen 64  scopeid 0x20<link>
        inet6 2001:468:c80:2129:21a:a0ff:febe:62c0  prefixlen 64
scopeid 0x0<global>
        ether 00:1a:a0:be:62:c0  txqueuelen 1000  (Ethernet)
        RX packets 2033321  bytes 346639175 (330.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 114047  bytes 17840888 (17.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 10078  bytes 698299 (681.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10078  bytes 698299 (681.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

== eu-readelf -n /boot/vmlinuz-3.12.6-200.fc19.i686.PAE | grep Build ==
    Build ID: 96999e758723db364228d675baf39ecd4aaaa491
== hexdump -C /sys/kernel/notes ==
00000000  04 00 00 00 06 00 00 00  06 00 00 00 58 65 6e 00  |............Xen.|
00000010  6c 69 6e 75 78 00 00 00  04 00 00 00 04 00 00 00  |linux...........|
00000020  07 00 00 00 58 65 6e 00  32 2e 36 00 04 00 00 00  |....Xen.2.6.....|
00000030  08 00 00 00 05 00 00 00  58 65 6e 00 78 65 6e 2d  |........Xen.xen-|
00000040  33 2e 30 00 04 00 00 00  04 00 00 00 03 00 00 00  |3.0.............|
00000050  58 65 6e 00 00 00 00 c0  04 00 00 00 04 00 00 00  |Xen.............|
00000060  01 00 00 00 58 65 6e 00  4c 52 ca c0 04 00 00 00  |....Xen.LR......|
00000070  04 00 00 00 02 00 00 00  58 65 6e 00 00 10 40 c0  |........Xen...@.|
00000080  04 00 00 00 2a 00 00 00  0a 00 00 00 58 65 6e 00  |....*.......Xen.|
00000090  21 77 72 69 74 61 62 6c  65 5f 70 61 67 65 5f 74  |!writable_page_t|
000000a0  61 62 6c 65 73 7c 70 61  65 5f 70 67 64 69 72 5f  |ables|pae_pgdir_|
000000b0  61 62 6f 76 65 5f 34 67  62 00 00 00 04 00 00 00  |above_4gb.......|
000000c0  04 00 00 00 09 00 00 00  58 65 6e 00 79 65 73 00  |........Xen.yes.|
000000d0  04 00 00 00 08 00 00 00  08 00 00 00 58 65 6e 00  |............Xen.|
000000e0  67 65 6e 65 72 69 63 00  04 00 00 00 10 00 00 00  |generic.........|
000000f0  0d 00 00 00 58 65 6e 00  01 00 00 00 00 00 00 00  |....Xen.........|
00000100  01 00 00 00 00 00 00 00  04 00 00 00 04 00 00 00  |................|
00000110  0e 00 00 00 58 65 6e 00  01 00 00 00 04 00 00 00  |....Xen.........|
00000120  04 00 00 00 0c 00 00 00  58 65 6e 00 00 00 80 f5  |........Xen.....|
00000130  04 00 00 00 04 00 00 00  04 00 00 00 58 65 6e 00  |............Xen.|
00000140  00 00 00 00 04 00 00 00  14 00 00 00 03 00 00 00  |................|
00000150  47 4e 55 00 96 99 9e 75  87 23 db 36 42 28 d6 75  |GNU....u.#.6B(.u|
00000160  ba f3 9e cd 4a aa a4 91                           |....J...|
00000168
== hexdump -C /sys/module/ata_generic/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  73 29 6b 93 ad 84 4e 83  b8 53 12 cb 74 f4 2f 78  |s)k...N..S..t./x|
00000020  5e b7 b4 0a                                       |^...|
00000024
== hexdump -C /sys/module/auth_rpcgss/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  d7 e0 e0 19 46 18 0c 23  6e cd 0b 7e ce eb 5f a6  |....F..#n..~.._.|
00000020  8b d3 08 c9                                       |....|
00000024
== hexdump -C /sys/module/binfmt_misc/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  ee 2b 15 d9 da e8 e8 ac  c5 21 fe 5d 22 6e 2d 85  |.+.......!.]"n-.|
00000020  10 3c bc 2f                                       |.<./|
00000024
== hexdump -C /sys/module/bluetooth/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  c0 9f 5e 3b a6 86 f9 55  21 17 5d eb 22 1d fb c6  |..^;...U!.]."...|
00000020  7b 58 3a 96                                       |{X:.|
00000024
== hexdump -C /sys/module/coretemp/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  b8 2b 6b a9 30 27 93 fe  80 50 11 a1 4c 58 30 72  |.+k.0'...P..LX0r|
00000020  49 5d 12 b9                                       |I]..|
00000024
== hexdump -C /sys/module/crc_itu_t/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  0f 65 71 36 46 3c 70 88  c9 d0 ff c4 7b 43 27 95  |.eq6F<p.....{C'.|
00000020  de da 09 a3                                       |....|
00000024
== hexdump -C /sys/module/dcdbas/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  34 98 08 8f 5a 4e 80 69  0e a6 58 df 6c fb 78 d2  |4...ZN.i..X.l.x.|
00000020  66 6f a3 3b                                       |fo.;|
00000024
== hexdump -C /sys/module/drm_kms_helper/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  be 82 02 7a 08 56 b3 3e  29 48 e1 4f c9 a9 8d 1e  |...z.V.>)H.O....|
00000020  5d de 44 b6                                       |].D.|
00000024
== hexdump -C /sys/module/drm/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  ed 60 82 f9 24 48 b9 d2  ba 9b 12 2e 6d 33 a6 c6  |.`..$H......m3..|
00000020  46 36 6a 1b                                       |F6j.|
00000024
== hexdump -C /sys/module/edac_core/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  5a 2e 4a c1 d0 f4 e6 58  86 22 df de 72 9e e4 1f  |Z.J....X."..r...|
00000020  e1 df ac 5d                                       |...]|
00000024
== hexdump -C /sys/module/firewire_core/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  6a 7d 70 0c 2d 41 14 b6  78 7e 27 95 84 64 f8 a4  |j}p.-A..x~'..d..|
00000020  11 72 82 0f                                       |.r..|
00000024
== hexdump -C /sys/module/firewire_ohci/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  fc dd 67 cd 35 1f 7e 0c  0b f1 00 88 3f e7 0d 12  |..g.5.~.....?...|
00000020  24 e2 9c 73                                       |$..s|
00000024
== hexdump -C /sys/module/i2c_algo_bit/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  7a 4c 48 52 bc ef a7 59  35 72 a8 70 f0 0d 1d 76  |zLHR...Y5r.p...v|
00000020  fa 75 5f 2a                                       |.u_*|
00000024
== hexdump -C /sys/module/i2c_core/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  44 56 25 4d ea 86 c6 0c  13 f9 de f5 1b d9 8c 6d  |DV%M...........m|
00000020  4e d7 98 ad                                       |N...|
00000024
== hexdump -C /sys/module/i2c_i801/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  23 e2 a9 34 ca ff e8 75  0c e0 74 7f a6 47 f2 a0  |#..4...u..t..G..|
00000020  ef 66 dc 69                                       |.f.i|
00000024
== hexdump -C /sys/module/i5000_edac/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  5e 70 62 76 6c d5 30 8d  29 7c 23 60 9c 67 88 cb  |^pbvl.0.)|#`.g..|
00000020  61 3f 5d 1a                                       |a?].|
00000024
== hexdump -C /sys/module/i5k_amb/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  77 28 bd 50 a5 de 9d fe  e8 54 2c 32 50 c4 4e 72  |w(.P.....T,2P.Nr|
00000020  31 b7 95 c8                                       |1...|
00000024
== hexdump -C /sys/module/inet_diag/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  39 c0 13 c5 f2 d1 05 cc  4f 83 d4 0d 37 b5 54 66  |9.......O...7.Tf|
00000020  21 8d 1b dc                                       |!...|
00000024
== hexdump -C /sys/module/ip6table_filter/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  04 db c6 eb 2b 9b 36 e8  58 fa f1 0d 1d c9 2f 69  |....+.6.X...../i|
00000020  0f 77 16 01                                       |.w..|
00000024
== hexdump -C /sys/module/ip6_tables/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  3e e8 e6 57 ea cc b4 19  1c 1b 3e 97 e5 c7 57 f9  |>..W......>...W.|
00000020  bc dc e8 2a                                       |...*|
00000024
== hexdump -C /sys/module/iTCO_vendor_support/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  0c 2e 6c 80 d6 65 a9 e9  a7 48 41 30 53 4a 36 f9  |..l..e...HA0SJ6.|
00000020  29 0f 13 4b                                       |)..K|
00000024
== hexdump -C /sys/module/iTCO_wdt/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  10 e8 68 a3 6b 2d 34 61  1f 59 86 da 37 83 26 61  |..h.k-4a.Y..7.&a|
00000020  54 d0 17 1a                                       |T...|
00000024
== hexdump -C /sys/module/kvm_intel/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  39 7c 96 c8 a7 02 1a fd  23 f3 a9 90 3b 28 09 81  |9|......#...;(..|
00000020  1e cb db 47                                       |...G|
00000024
== hexdump -C /sys/module/kvm/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  2a a9 4f 75 3b 1f 03 f5  8c 00 18 7d 55 aa 4a 35  |*.Ou;......}U.J5|
00000020  e1 b1 1c bf                                       |....|
00000024
== hexdump -C /sys/module/lockd/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  04 4e 22 84 c8 a5 a7 e9  dd eb 04 01 15 87 73 de  |.N"...........s.|
00000020  ac 6f ef 01                                       |.o..|
00000024
== hexdump -C /sys/module/lpc_ich/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  8b 64 14 85 45 27 31 1d  d6 ce 60 af e0 52 fe 81  |.d..E'1...`..R..|
00000020  5a 58 e6 75                                       |ZX.u|
00000024
== hexdump -C /sys/module/mfd_core/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  43 4b 78 55 7f 45 97 95  a3 06 31 ee 55 cc 17 e9  |CKxU.E....1.U...|
00000020  f7 b7 b9 3c                                       |...<|
00000024
== hexdump -C /sys/module/microcode/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  96 57 b0 24 66 5b b0 70  0c 96 59 4c fc 87 59 99  |.W.$f[.p..YL..Y.|
00000020  d8 f7 0f 34                                       |...4|
00000024
== hexdump -C /sys/module/mxm_wmi/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  2c 78 4c 82 54 6d 9d 2a  30 84 56 a2 07 64 f7 06  |,xL.Tm.*0.V..d..|
00000020  28 3e 42 bb                                       |(>B.|
00000024
== hexdump -C /sys/module/nf_conntrack_ipv4/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  a2 59 c6 54 5b 7c 9f 58  6e 6b 5c f3 93 85 53 e9  |.Y.T[|.Xnk\...S.|
00000020  26 0d 40 65                                       |&.@e|
00000024
== hexdump -C /sys/module/nf_conntrack/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  db 3a a3 06 0d 2b 48 06  a8 96 a7 3a 92 8a 3d 5c  |.:...+H....:..=\|
00000020  98 dc 5c 19                                       |..\.|
00000024
== hexdump -C /sys/module/nf_defrag_ipv4/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  ca ac 11 32 92 6f 16 1d  fd 23 70 9a 3b b8 1c 65  |...2.o...#p.;..e|
00000020  60 0e 2c 97                                       |`.,.|
00000024
== hexdump -C /sys/module/nfnetlink_log/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  31 11 13 79 db 60 08 4a  a1 29 0b 3d c8 98 df 5a  |1..y.`.J.).=...Z|
00000020  6b a9 b9 95                                       |k...|
00000024
== hexdump -C /sys/module/nfnetlink/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  f1 42 66 59 68 5d f1 02  40 dd ad be 27 06 52 4f  |.BfYh]..@...'.RO|
00000020  eb 03 63 c0                                       |..c.|
00000024
== hexdump -C /sys/module/nfnetlink_queue/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  f6 3b b1 a2 f0 3d 5d f1  ab 0f 5b 39 f1 13 11 7b  |.;...=]...[9...{|
00000020  6a 4d 88 de                                       |jM..|
00000024
== hexdump -C /sys/module/nfs_acl/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  90 42 38 8c fc 5c 10 01  16 c9 89 18 1d e4 91 5f  |.B8..\........._|
00000020  ba 88 10 31                                       |...1|
00000024
== hexdump -C /sys/module/nfsd/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  b4 4c 86 94 14 2a ac 13  e2 8c db 80 a7 3c 57 c2  |.L...*.......<W.|
00000020  6e 1b 4d b8                                       |n.M.|
00000024
== hexdump -C /sys/module/nouveau/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  eb cd 26 24 8f 92 87 bf  72 3a 05 26 57 25 23 bd  |..&$....r:.&W%#.|
00000020  60 e4 c1 8a                                       |`...|
00000024
== hexdump -C /sys/module/parport/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  5e db 30 58 5d 08 e9 26  d2 44 6a 83 0b db 83 bf  |^.0X]..&.Dj.....|
00000020  9b 7a 91 ea                                       |.z..|
00000024
== hexdump -C /sys/module/parport_pc/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  90 91 b9 9f bb 2f c1 97  04 f8 39 4c 7b 44 ba f0  |...../....9L{D..|
00000020  3e f3 e2 67                                       |>..g|
00000024
== hexdump -C /sys/module/pata_acpi/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  96 f6 e8 88 94 03 a1 80  7f a2 d1 25 ce a4 c7 a3  |...........%....|
00000020  33 88 e2 da                                       |3...|
00000024
== hexdump -C /sys/module/ppdev/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  ea bc f4 32 f4 1a c8 86  c1 d9 e5 06 12 b3 1e 54  |...2...........T|
00000020  ea b6 60 79                                       |..`y|
00000024
== hexdump -C /sys/module/pps_core/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  64 89 7a 30 5e 24 54 0c  90 4b 07 fe 0e 90 2f 2e  |d.z0^$T..K..../.|
00000020  47 0a 40 89                                       |G.@.|
00000024
== hexdump -C /sys/module/ptp/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  07 f5 3e c1 e7 a6 dd 22  28 01 42 13 b5 a5 ba 6b  |..>...."(.B....k|
00000020  70 90 36 4c                                       |p.6L|
00000024
== hexdump -C /sys/module/rfkill/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  f3 32 ca c1 db 06 b0 a0  e1 54 7a c4 68 b3 86 0b  |.2.......Tz.h...|
00000020  de 8e 25 47                                       |..%G|
00000024
== hexdump -C /sys/module/serio_raw/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  83 15 39 ed f6 6b 50 18  28 0e f0 46 dc 1c ce 40  |..9..kP.(..F...@|
00000020  7b 2b e1 cd                                       |{+..|
00000024
== hexdump -C /sys/module/snd_hda_codec_idt/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  62 99 6f 95 da d7 85 4b  98 40 85 a2 7a 86 2c 48  |b.o....K.@..z.,H|
00000020  2b b5 18 f6                                       |+...|
00000024
== hexdump -C /sys/module/snd_hda_codec/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  81 f0 57 af 00 6e 47 d5  3d a3 17 80 00 cf 0e 3f  |..W..nG.=......?|
00000020  24 44 fa 39                                       |$D.9|
00000024
== hexdump -C /sys/module/snd_hda_intel/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  ca da 8c 16 53 bd 71 fc  e7 d6 96 09 cc 3a e8 f7  |....S.q......:..|
00000020  99 74 c9 7d                                       |.t.}|
00000024
== hexdump -C /sys/module/snd_hwdep/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  3c 6d db 9c 59 64 4e 47  66 67 fa c0 4a bb d9 fd  |<m..YdNGfg..J...|
00000020  ac 86 fd 0d                                       |....|
00000024
== hexdump -C /sys/module/snd/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  f4 04 16 6d 22 fb c8 10  c1 ae 09 a3 37 15 05 7f  |...m".......7...|
00000020  67 46 55 c4                                       |gFU.|
00000024
== hexdump -C /sys/module/snd_page_alloc/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  29 78 e0 ee bb f9 7b 24  54 51 fd 95 05 e8 53 91  |)x....{$TQ....S.|
00000020  ea c2 0c df                                       |....|
00000024
== hexdump -C /sys/module/snd_pcm/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  b7 90 7c ce f8 f2 e4 0c  aa 8f a0 c6 89 b0 f3 89  |..|.............|
00000020  9c 58 8a 43                                       |.X.C|
00000024
== hexdump -C /sys/module/snd_seq_device/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  1b 75 01 4d c4 fd f6 24  62 ba 76 5a 40 05 68 b2  |.u.M...$b.vZ@.h.|
00000020  7d 51 33 16                                       |}Q3.|
00000024
== hexdump -C /sys/module/snd_seq/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  0f f6 36 d0 4f e4 a7 86  b4 cd dc 37 17 08 56 1b  |..6.O......7..V.|
00000020  8b 9c 25 0d                                       |..%.|
00000024
== hexdump -C /sys/module/snd_timer/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  ac 24 33 9a ca 51 ce 66  16 fb 21 2b c9 97 d6 6e  |.$3..Q.f..!+...n|
00000020  2c b8 00 06                                       |,...|
00000024
== hexdump -C /sys/module/soundcore/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  3b 7c c2 4f 69 a6 0b 86  f6 76 0d 1e c5 71 68 33  |;|.Oi....v...qh3|
00000020  59 cb 9c b4                                       |Y...|
00000024
== hexdump -C /sys/module/sunrpc/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  78 fd de e7 1f 32 59 22  4f 60 f6 d0 d6 1b b2 2a  |x....2Y"O`.....*|
00000020  57 d8 97 93                                       |W...|
00000024
== hexdump -C /sys/module/tcp_diag/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  c0 49 20 01 27 92 f1 16  56 9b 3c 60 ac cd dd 2c  |.I .'...V.<`...,|
00000020  b0 15 eb 9d                                       |....|
00000024
== hexdump -C /sys/module/tg3/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  ed 55 e9 c6 3a 06 63 13  cd bd a4 95 2d f9 97 fb  |.U..:.c.....-...|
00000020  bd a8 07 8d                                       |....|
00000024
== hexdump -C /sys/module/ttm/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  c1 e4 f8 ab 9e ab a9 23  9f cb f2 86 c0 47 bc 86  |.......#.....G..|
00000020  00 4c 71 99                                       |.Lq.|
00000024
== hexdump -C /sys/module/udp_diag/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  24 24 e5 4c 4d 23 b4 80  66 45 05 8c 5c 73 dc 92  |$$.LM#..fE..\s..|
00000020  a2 93 8c 01                                       |....|
00000024
== hexdump -C /sys/module/video/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  ba 9d 90 6a f5 ae 9e a9  03 9d 2f 8b 47 f7 e6 cb  |...j....../.G...|
00000020  4a b1 be 35                                       |J..5|
00000024
== hexdump -C /sys/module/wmi/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  3b 26 96 2e 39 d8 a7 ac  0d 67 94 1d d6 cc c8 20  |;&..9....g..... |
00000020  0f a8 04 df                                       |....|
00000024
== hexdump -C /sys/module/xt_conntrack/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  c6 ce 70 08 ad 30 23 04  c5 49 92 63 41 3f e2 d5  |..p..0#..I.cA?..|
00000020  a5 f3 64 5a                                       |..dZ|
00000024
== hexdump -C /sys/module/xt_recent/notes/.note.gnu.build-id ==
00000000  04 00 00 00 14 00 00 00  03 00 00 00 47 4e 55 00  |............GNU.|
00000010  44 5d bd 6b d3 57 e8 47  ad b8 62 e4 73 88 a0 98  |D].k.W.G..b.s...|
00000020  72 a7 f9 68                                       |r..h|
00000024
== getenforce ==
Enforcing
== NSS ==
NSS database path '/root/.systemtap/ssl/client' does not exist
NSS database path '/root/.systemtap/ssl/server' does not exist
NSS database path '/etc/systemtap/ssl/server' does not exist
NSS database path '/etc/systemtap/staprun' does not exist

Thanks,
Xiaokui

On Fri, Jan 10, 2014 at 1:46 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Xiaokui Shu <subbyte@gmail.com> writes:
>
>> [...] I start systemtap with a simple probe
>> (shown below), I get two warnings in Pass 5. While running systemtap,
>> [...]
>> WARNING: probe process("/usr/sbin/sendmail.sendmail").function("mime_getchar_crlf.part.1")
>> inode-offset 0005e620 registration error (rc -524)
>> WARNING: probe process("/usr/sbin/sendmail.sendmail").function("frame_dummy")
>> inode-offset 0001cc80 registration error (rc -524)
>
> That's -ENOTSUPP; maybe there is some sort of instruction in there the
> kernel doesn't like.
>
>> I can send emails to the monitored host, but my mailbox on the host
>> does not receive them. [...]  Can anyone help me to figure out what
>> is the problem brought by the tracing?
>
> This sounds should not happen.
>
>> probe process(@1).function("*")
>> {
>>     printf ("%s:0x%x\n", probefunc (), ustack (1))
>> }
>
> You might try rerunning '# stap -DSTP_ALIBI foo.stp' to eliminate most
> of the systemtap probe handler logic out of the script.  If the
> userspace sendmail process still misbehaves, this helps incriminate
> uprobes -- and vice versa.
>
> Could you collect kernel / stap version information, or the whole
> `stap-report` shebang?
>
>
> - FChE

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: sendmail function probe registration error with systemtap
       [not found]   ` <CAMU3N_KaOPbsYQS1io=_v-g4PzzQ4mAbk0GAOveN2tf-dLmaCg@mail.gmail.com>
  2014-01-10 20:27     ` Fwd: " Xiaokui Shu
@ 2014-01-10 21:15     ` Frank Ch. Eigler
  2014-01-10 21:43       ` Xiaokui Shu
  1 sibling, 1 reply; 6+ messages in thread
From: Frank Ch. Eigler @ 2014-01-10 21:15 UTC (permalink / raw)
  To: Xiaokui Shu; +Cc: systemtap

On Fri, Jan 10, 2014 at 03:26:09PM -0500, Xiaokui Shu wrote:
> I find nothing changed with -DSTP_ALIBI. It does not stop sendmail
> from not receiving emails. Do you suggest this is a uprobe problem
> instead of systemtap?

Yup, high likelihood.

> The command I run stap is:
> #stap -DSTP_ALIBI -o sendmail.trace -v trace.stp sendmail

> == stap -V ==
> Systemtap translator/driver (version 2.4/0.156, rpm 2.4-1.fc19)
> [...]
> == dmesg | fgrep 'Linux version' ==
> [    0.000000] Linux version 3.12.6-200.fc19.i686.PAE
> (mockbuild@bkernel02) (gcc version 4.8.2 20131212 (Red Hat 4.8.2-7)
> (GCC) ) #1 SMP Mon Dec 23 16:58:40 UTC 2013
> [...]

Thanks a lot.  So it's a nice recent kernel & systemtap combination.

One way to proceed further is to try to reproduce the problem with
"perf probe" rather than systemtap.  For that, one might run "stap -k
..." to preserve the C intermediate code to fish out the low-level
probe addresses, arrange to feed the same set to perf-probe, then
activate them on a copy of sendmail, and see if it flies.  In case
it doesn't, it's time to file a kernel bugzilla entry.

We can help with this part, but would need e.g. the generated .c file
and the exact rpm version of your sendmail.


- FChE

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: sendmail function probe registration error with systemtap
  2014-01-10 21:15     ` Frank Ch. Eigler
@ 2014-01-10 21:43       ` Xiaokui Shu
  2014-01-10 23:10         ` Xiaokui Shu
  0 siblings, 1 reply; 6+ messages in thread
From: Xiaokui Shu @ 2014-01-10 21:43 UTC (permalink / raw)
  To: Frank Ch. Eigler; +Cc: systemtap

Dear Frank,

I am very glad that you can help. I am not familiar with perf probe,
and thank you for your kindness to help us.

I am running Fedora 19 with the latest kernel in the repository. I
installed sendmail from the official repository as well as debug-info
for glibc and coreutils. sendmail is tested working properly without
systemtap tracing scripts. We are doing experiments tracing function
calls of sendmail using trace.stp I provide in my first email.

# yum list installed | grep sendmail
sendmail.i686                         8.14.7-1.fc19                     @fedora
sendmail-cf.noarch                    8.14.7-1.fc19                     @fedora
sendmail-debuginfo.i686               8.14.7-1.fc19
 @fedora-debuginfo
sendmail-devel.i686                   8.14.7-1.fc19                     @fedora
sendmail-milter.i686                  8.14.7-1.fc19                     @fedora

# yum list installed | grep glibc
glibc.i686                            2.17-20.fc19                      @updates
glibc-common.i686                     2.17-20.fc19                      @updates
glibc-debuginfo.i686                  2.17-20.fc19
 @updates-debuginfo
glibc-debuginfo-common.i686           2.17-20.fc19
 @updates-debuginfo
glibc-devel.i686                      2.17-20.fc19                      @updates
glibc-headers.i686                    2.17-20.fc19                      @updates

# yum list installed | grep coreutils
coreutils.i686                        8.21-11.fc19                      @fedora
coreutils-debuginfo.i686              8.21-11.fc19
 @fedora-debuginfo
policycoreutils.i686                  2.1.14-46.6.fc19                  @updates


Best,
Xiaokui

On Fri, Jan 10, 2014 at 4:15 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> On Fri, Jan 10, 2014 at 03:26:09PM -0500, Xiaokui Shu wrote:
>> I find nothing changed with -DSTP_ALIBI. It does not stop sendmail
>> from not receiving emails. Do you suggest this is a uprobe problem
>> instead of systemtap?
>
> Yup, high likelihood.
>
>> The command I run stap is:
>> #stap -DSTP_ALIBI -o sendmail.trace -v trace.stp sendmail
>
>> == stap -V ==
>> Systemtap translator/driver (version 2.4/0.156, rpm 2.4-1.fc19)
>> [...]
>> == dmesg | fgrep 'Linux version' ==
>> [    0.000000] Linux version 3.12.6-200.fc19.i686.PAE
>> (mockbuild@bkernel02) (gcc version 4.8.2 20131212 (Red Hat 4.8.2-7)
>> (GCC) ) #1 SMP Mon Dec 23 16:58:40 UTC 2013
>> [...]
>
> Thanks a lot.  So it's a nice recent kernel & systemtap combination.
>
> One way to proceed further is to try to reproduce the problem with
> "perf probe" rather than systemtap.  For that, one might run "stap -k
> ..." to preserve the C intermediate code to fish out the low-level
> probe addresses, arrange to feed the same set to perf-probe, then
> activate them on a copy of sendmail, and see if it flies.  In case
> it doesn't, it's time to file a kernel bugzilla entry.
>
> We can help with this part, but would need e.g. the generated .c file
> and the exact rpm version of your sendmail.
>
>
> - FChE

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: sendmail function probe registration error with systemtap
  2014-01-10 21:43       ` Xiaokui Shu
@ 2014-01-10 23:10         ` Xiaokui Shu
  0 siblings, 0 replies; 6+ messages in thread
From: Xiaokui Shu @ 2014-01-10 23:10 UTC (permalink / raw)
  To: Frank Ch. Eigler; +Cc: systemtap

Hi Frank,

I have two things to add.

1) I use -DSTP_NO_OVERLOAD=1 to make sure the script does not quit. I
forget to write it in the previous email.
2) I find the following scenario about sendmail:
If I run stap to trace sendmail, no matter how many emails I send to
the monitored server, I can get only two processes for sendmail on the
server.

root      5323  0.0  0.0  14304  2532 ?        Ss   Jan08   0:06
sendmail: accepting connections
smmsp     5338  0.0  0.0  13852  2048 ?        Ss   Jan08   0:00
sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue

When I stop stap, I get more processes:

root      5323  0.0  0.0  14304  2532 ?        Ss   Jan08   0:06
sendmail: accepting connections
smmsp     5338  0.0  0.0  13852  2048 ?        Ss   Jan08   0:00
sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root     14757  0.0  0.0  14300  3328 ?        S    18:03   0:00
sendmail: ./s0AN3uP7014749 from queue
root     14759  0.0  0.0  14300  3328 ?        S    18:03   0:00
sendmail: ./s0AN3uis014755 from queue
...

Then I can receive emails in my mailbox. It seems that sendmail queued
requests and the tracing block sendmail from forking subprocesses to
handle the requests in the queue.

Best,
Xiaokui

On Fri, Jan 10, 2014 at 4:43 PM, Xiaokui Shu <subbyte@gmail.com> wrote:
> Dear Frank,
>
> I am very glad that you can help. I am not familiar with perf probe,
> and thank you for your kindness to help us.
>
> I am running Fedora 19 with the latest kernel in the repository. I
> installed sendmail from the official repository as well as debug-info
> for glibc and coreutils. sendmail is tested working properly without
> systemtap tracing scripts. We are doing experiments tracing function
> calls of sendmail using trace.stp I provide in my first email.
>
> # yum list installed | grep sendmail
> sendmail.i686                         8.14.7-1.fc19                     @fedora
> sendmail-cf.noarch                    8.14.7-1.fc19                     @fedora
> sendmail-debuginfo.i686               8.14.7-1.fc19
>  @fedora-debuginfo
> sendmail-devel.i686                   8.14.7-1.fc19                     @fedora
> sendmail-milter.i686                  8.14.7-1.fc19                     @fedora
>
> # yum list installed | grep glibc
> glibc.i686                            2.17-20.fc19                      @updates
> glibc-common.i686                     2.17-20.fc19                      @updates
> glibc-debuginfo.i686                  2.17-20.fc19
>  @updates-debuginfo
> glibc-debuginfo-common.i686           2.17-20.fc19
>  @updates-debuginfo
> glibc-devel.i686                      2.17-20.fc19                      @updates
> glibc-headers.i686                    2.17-20.fc19                      @updates
>
> # yum list installed | grep coreutils
> coreutils.i686                        8.21-11.fc19                      @fedora
> coreutils-debuginfo.i686              8.21-11.fc19
>  @fedora-debuginfo
> policycoreutils.i686                  2.1.14-46.6.fc19                  @updates
>
>
> Best,
> Xiaokui
>
> On Fri, Jan 10, 2014 at 4:15 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
>> On Fri, Jan 10, 2014 at 03:26:09PM -0500, Xiaokui Shu wrote:
>>> I find nothing changed with -DSTP_ALIBI. It does not stop sendmail
>>> from not receiving emails. Do you suggest this is a uprobe problem
>>> instead of systemtap?
>>
>> Yup, high likelihood.
>>
>>> The command I run stap is:
>>> #stap -DSTP_ALIBI -o sendmail.trace -v trace.stp sendmail
>>
>>> == stap -V ==
>>> Systemtap translator/driver (version 2.4/0.156, rpm 2.4-1.fc19)
>>> [...]
>>> == dmesg | fgrep 'Linux version' ==
>>> [    0.000000] Linux version 3.12.6-200.fc19.i686.PAE
>>> (mockbuild@bkernel02) (gcc version 4.8.2 20131212 (Red Hat 4.8.2-7)
>>> (GCC) ) #1 SMP Mon Dec 23 16:58:40 UTC 2013
>>> [...]
>>
>> Thanks a lot.  So it's a nice recent kernel & systemtap combination.
>>
>> One way to proceed further is to try to reproduce the problem with
>> "perf probe" rather than systemtap.  For that, one might run "stap -k
>> ..." to preserve the C intermediate code to fish out the low-level
>> probe addresses, arrange to feed the same set to perf-probe, then
>> activate them on a copy of sendmail, and see if it flies.  In case
>> it doesn't, it's time to file a kernel bugzilla entry.
>>
>> We can help with this part, but would need e.g. the generated .c file
>> and the exact rpm version of your sendmail.
>>
>>
>> - FChE

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2014-01-10 23:10 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-01-09 23:12 sendmail function probe registration error with systemtap Xiaokui Shu
2014-01-10 18:46 ` Frank Ch. Eigler
     [not found]   ` <CAMU3N_KaOPbsYQS1io=_v-g4PzzQ4mAbk0GAOveN2tf-dLmaCg@mail.gmail.com>
2014-01-10 20:27     ` Fwd: " Xiaokui Shu
2014-01-10 21:15     ` Frank Ch. Eigler
2014-01-10 21:43       ` Xiaokui Shu
2014-01-10 23:10         ` Xiaokui Shu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).