[Bug tapsets/19300] New: Many SystemTap tapsets failing on kernel.org 4.2.0 kernel

[Bug tapsets/19300] New: Many SystemTap tapsets failing on kernel.org 4.2.0 kernel

[steven_b_wood at fastmail dot fm]

https://sourceware.org/bugzilla/show_bug.cgi?id=19300

            Bug ID: 19300
           Summary: Many SystemTap tapsets failing on kernel.org 4.2.0
                    kernel
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: tapsets
          Assignee: systemtap at sourceware dot org
          Reporter: steven_b_wood at fastmail dot fm
  Target Milestone: ---

Created attachment 8816
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8816&action=edit
Configuration Files and Test Results

Hello,

I recently installed SystemTap on a vanilla 4.2.0 kernel(Linux From Scratch
7.8). While going through the SystemTap Beginners Guide and testing the various
scripts I notice that some tapsets work but unfortunately the network related
tapsets -- the ones I want to use -- do not. 

For example, running a tapset like:

probe tcp.ipv4.receive {
  printf(" %15s %15s  %5d  %5d  %d  %d  %d  %d  %d  %d\n",
         saddr, daddr, sport, dport, urg, ack, psh, rst, syn, fin)
}

will result in:

semantic error: while processing function __tcp_skb_ack
   thrown from: elaborate.cxx:4955
semantic error: type definition 'tcphdr' not found in 'kernel': operator
'@cast' at /usr/share/systemtap/tapset/linux/tcp.stp:121:9
   thrown from: elaborate.cxx:5541
        source:         return @cast(tcphdr, "tcphdr")->ack

I know that the necessary headers are there on my box and that the kernel has
been compiled with CONFIG_DEBUG_INFO and the other required CONFIG settings.

For some reason wildcards don't work either -- not a big deal to me but perhaps
it may illuminate what is going on.

stap -vv -e 'probe kernel.function("*@net/socket.c") { }'

semantic error: while resolving probe point: identifier 'kernel' at <input>:1:7
   thrown from: elaborate.cxx:1069
        source: probe kernel.function("*@net/socket.c") { }
                      ^

As an example of what works -- probes like these work great:

probe syscall.open
{
  printf ("%s(%d) open\n", execname(), pid())
}

--or--

global count_jiffies, count_ms
probe timer.jiffies(100) { count_jiffies ++ }
probe timer.ms(100) { count_ms ++ }
probe timer.ms(12345)
{
  hz=(1000*count_jiffies) / count_ms
  printf ("jiffies:ms ratio %d:%d => CONFIG_HZ=%d\n",
    count_jiffies, count_ms, hz)
  exit ()
}

I am hoping that the root cause of these issues is perhaps configuration as
each Linux distro puts things in slightly different places. SystemTap seems
pretty impressive and I hope to be able to fully use it in the future on
kernel.org kernels, so any help on this would be greatly appreciated.

-------ATTACHMENTS-------
1) kernel .config showing modules for systemtap installed per instructions in
'Building a kernel.org kernel' under Installation -> Build Your Own
2) elfutils 0.164 config.log
3) systemtap 2.9 config.log
4) results of make installcheck
5) successful results of stap -vv variables-in-printf.stp
6) failure results of stap -vv tcpdumplike.stp
7) successful results of stap -vv timer-jiffies.stp
8) failure results of stap -vv wildcards.stp

-------SYSTEMTAP VERSION-------
Systemtap translator/driver (version 2.9/0.164, non-git sources)
Copyright (C) 2005-2015 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: NLS TR1_UNORDERED_MAP

-------KERNEL VERSION------
Linux xyz_lfs 4.2.0 #1 SMP Thu Nov 26 22:12:49 PST 2015 i686 GNU/Linux

------DEBUGINFO PATH------- (as per 'Building a kernel.org kernel')
lrwxrwxrwx 1 root root 18 Nov 21 15:36 /lib/modules/4.2.0/build ->
/sources/linux-4.2

-------GCC VERSION------
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/i686-pc-linux-gnu/5.2.0/lto-wrapper
Target: i686-pc-linux-gnu
Configured with: ../gcc-5.2.0/configure --prefix=/usr --enable-languages=c,c++
--disable-multilib --disable-bootstrap --with-system-zlib
Thread model: posix
gcc version 5.2.0 (GCC)

------ELFUTILS VERSION------
elfutils-0.164

-------ENVIRONMENT VARIABLES------
HZ=100
SHELL=/bin/bash
TERM=xterm-256color
USER=root
MAIL=/var/mail/root
PATH=/sbin:/bin:/usr/sbin:/usr/bin
PWD=/sources/linux-4.2
LANG=en_US.utf8#
SHLVL=1
HOME=/root
LOGNAME=root
_=/usr/bin/env
OLDPWD=/usr/share/systemtap/tapset/linux

------ALL SOFTWARE ON BOX------
-su-4.3# ls /sources
acl-2.2.52.src.tar.gz               gettext-0.19.5.1.tar.xz              
ncurses-6.0.tar.gz
attr-2.4.47.src.tar.gz              git-2.6.0.tar.xz                     
openssh-7.1p1.tar.gz
autoconf-2.69.tar.xz                glibc-2.22                           
openssl-1.0.2d.tar.gz
automake-1.15.tar.xz                glibc-2.22-fhs-1.patch               
patch-2.7.5.tar.xz
bash-4.3.30.tar.gz                  glibc-2.22.tar.xz                    
perl-5.22.0.tar.bz2
bash-4.3.30-upstream_fixes-2.patch  glibc-2.22-upstream_i386_fix-1.patch 
pkg-config-0.28.tar.gz
bc-1.06.95-memory_leak-1.patch      gmp-6.0.0a.tar.xz                    
procps-ng-3.3.11.tar.xz
bc-1.06.95.tar.bz2                  gperf-3.0.4.tar.gz                   
psmisc-22.21.tar.gz
binutils-2.25.1.tar.bz2             grep-2.21.tar.xz                     
readline-6.3.tar.gz
bison-3.0.4.tar.xz                  groff-1.22.3.tar.gz                  
readline-6.3-upstream_fixes-3.patch
blfs-bootscripts-20150924           grub-2.02~beta2.tar.xz               
sed-4.2.2.tar.bz2
blfs-bootscripts-20150924.tar.bz2   gzip-1.6.tar.xz                      
shadow-4.2.1.tar.xz
bzip2-1.0.6-install_docs-1.patch    iana-etc-2.30.tar.bz2                
strace-4.10.tar.xz
bzip2-1.0.6.tar.gz                  inetutils-1.9.4.tar.xz               
sysklogd-1.5.1.tar.gz
check-0.10.0.tar.gz                 intltool-0.51.0.tar.gz               
systemtap-2.9
config                              iproute2-4.2.0.tar.xz                
systemtap-2.9.tar.gz
coreutils-8.24-i18n-1.patch         kbd-2.0.3-backspace-1.patch          
sysvinit-2.88dsf-consolidated-1.patch
coreutils-8.24.tar.xz               kbd-2.0.3.tar.xz                     
sysvinit-2.88dsf.tar.bz2
curl-7.45.0.tar.lzma                kmod-21.tar.xz                       
tar-1.28.tar.xz
dejagnu-1.5.3.tar.gz                less-458.tar.gz                      
tcl-core8.6.4-src.tar.gz
diffutils-3.3.tar.xz                lfs-bootscripts-20150222.tar.bz2     
texinfo-6.0.tar.xz
e2fsprogs-1.42.13.tar.gz            libcap-2.24.tar.xz                   
tools.tar
elfutils-0.164                      libpipeline-1.4.1.tar.gz             
trace-cmd
elfutils-0.164.tar.bz2              libtool-2.4.6.tar.xz                 
tzdata2015f.tar.gz
eudev-3.1.2.tar.gz                  linux-4.2                            
udev-lfs-20140408.tar.bz2
expat-2.1.0.tar.gz                  linux-4.2.tar.xz                     
util-linux-2.27.tar.xz
expect5.45.tar.gz                   m4-1.4.17.tar.xz                     
vim-7.4.tar.bz2
file-5.24.tar.gz                    make-4.1.tar.bz2                     
wget-1.16.3.tar.xz
findutils-4.4.2.tar.gz              man-db-2.7.2.tar.xz                  
which-2.21.tar.gz
flex-2.5.39.tar.xz                  man-pages-4.02.tar.xz                
XML-Parser-2.44.tar.gz
gawk-4.1.3.tar.xz                   md5sums                              
xz-5.2.1.tar.xz
gcc-5.2.0.tar.bz2                   mpc-1.0.3.tar.gz                     
zlib-1.2.8.tar.xz
gdb-7.10.tar.xz                     mpfr-3.1.3.tar.xz
gdbm-1.11.tar.gz                    mpfr-3.1.3-upstream_fixes-1.patch

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/19300] Many SystemTap tapsets failing on kernel.org 4.2.0 kernel

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19300

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
The definition of that tcphdr type is read out of the kernel vmlinux image (if
the tapset doesn't request a header-file based cast, like e.g. inet_sock on
tapset/linux/tcp.stp line 80.  The problematic @cast(... "tcphdr") bits could
be converted to @cast(... "tcphdr", "kernel<net/tcp.h>") or similar.  But if
debuginfo is missing, other things will be broken too.

Could you forward the stap-report output, as requested by the [man
error::pass4] and [man error::reporting] pages?  I wouldn't be surprised if the
vmlinux file was not as advertised.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/19300] Many SystemTap tapsets failing on kernel.org 4.2.0 kernel

[steven_b_wood at fastmail dot fm]

https://sourceware.org/bugzilla/show_bug.cgi?id=19300

--- Comment #2 from SBW <steven_b_wood at fastmail dot fm> ---
Created attachment 8817
  --> https://sourceware.org/bugzilla/attachment.cgi?id=8817&action=edit
stap-report

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/19300] Many SystemTap tapsets failing on kernel.org 4.2.0 kernel

[steven_b_wood at fastmail dot fm]

https://sourceware.org/bugzilla/show_bug.cgi?id=19300

--- Comment #3 from SBW <steven_b_wood at fastmail dot fm> ---
I'm using a VM so I redid installing elfutils/systemtap on my stock image. 

The tapset problems relating to networking are the same. 

I uploaded the stap-report output.

I also tried another test:
-su-4.3# cat tcp_connections.stp
probe begin {
  printf("%6s %16s %6s %6s %16s\n",
         "UID", "CMD", "PID", "PORT", "IP_SOURCE")
}

probe kernel.function("tcp_accept").return?,
      kernel.function("inet_csk_accept").return? {
  sock = $return
  if (sock != 0)
    printf("%6d %16s %6d %6d %16s\n", uid(), execname(), pid(),
           inet_get_local_port(sock), inet_get_ip_source(sock))
}

And got these errors:

semantic error: while processing function __ip_sock_daddr
   thrown from: elaborate.cxx:4955
semantic error: unable to find member 'inet' for struct inet_sock
(alternatives: pinet6, inet_id, sk, hdrincl, mc_index, min_ttl, tos, uc_index,
cork, inet_opt, mc_list, freebind, is_icsk, mc_ttl, uc_ttl, mc_all, recverr,
mc_addr, mc_loop, pmtudisc, rcv_tos, inet_saddr, inet_sport, nodefrag,
transparent, rx_dst_ifindex, cmsg_flags, convert_csum, bind_address_no_port):
operator '->' at /usr/share/systemtap/tapset/linux/ip.stp:107:48
   thrown from: dwflpp.cxx:3600
        source:                         @cast(sock, "inet_sock",
"kernel<net/ip.h>")->inet->daddr)))
                                                                               
    ^
        in expansion of macro: operator '@alternate' at
/usr/share/systemtap/tapset/choose_defined.stpm:3:57
        source:         ( @defined(@value_if_defined) ? (@value_if_defined) :
(@alternate) )
                                                                              
^
        in expansion of macro: operator '@choose_defined' at
/usr/share/systemtap/tapset/linux/ip.stp:106:7
        source:                     @choose_defined(@cast(sock,
"inet_sock")->daddr, # kernel >= 2.6.11
                                    ^
        in expansion of macro: operator '@alternate' at
/usr/share/systemtap/tapset/choose_defined.stpm:3:57
        source:         ( @defined(@value_if_defined) ? (@value_if_defined) :
(@alternate) )
                                                                              
^
        in expansion of macro: operator '@choose_defined' at
/usr/share/systemtap/tapset/linux/ip.stp:105:3
        source:                 @choose_defined(@cast(sock,
"inet_sock")->inet_daddr, # kernel >= 2.6.33
                                ^
        in expansion of macro: operator '@alternate' at
/usr/share/systemtap/tapset/choose_defined.stpm:3:57
        source:         ( @defined(@value_if_defined) ? (@value_if_defined) :
(@alternate) )
                                                                              
^
        in expansion of macro: operator '@choose_defined' at
/usr/share/systemtap/tapset/linux/ip.stp:104:9
        source:         return @choose_defined(@cast(sock,
"inet_sock")->sk->__sk_common->skc_daddr, # kernel >= 2.6.38
                               ^

Pass 2: analyzed script: 2 probe(s), 11 function(s), 4 embed(s), 0 global(s)
using 44492virt/40256res/6408shr/34060data kb, in 1220usr/2330sys/3530real ms.
Pass 2: analysis failed.  [man error::pass2]

This behavior is the same as an earlier bug:

https://sourceware.org/bugzilla/show_bug.cgi?format=multiple&id=15171

Perhaps the above error indicates that stap can't determine my kernel version?
(Some variable not set? I do know uname works fine).

Regardless I can still use some of the probes. If anything obvious is wrong
please let me know.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/19300] Many SystemTap tapsets failing on kernel.org 4.2.0 kernel

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19300

--- Comment #4 from Frank Ch. Eigler <fche at redhat dot com> ---
Interesting; the code works here on on f22 against 4.2.6.
I wonder if the problem relates to lack of kernel debuginfo.
The tapset has this construct:

function __ip_sock_daddr:long (sock:long)
        return @choose_defined(@cast(sock,
"inet_sock")->sk->__sk_common->skc_daddr, # kernel >= 2.6.38
                @choose_defined(@cast(sock, "inet_sock")->inet_daddr, 
                    @choose_defined(@cast(sock, "inet_sock")->daddr,
                        @cast(sock, "inet_sock",
"kernel<net/ip.h>")->inet->daddr)))

Note how only the last @cast() tries to generate debuginfo for
the struct type via a @cast(var, "type", "kernel<FOO.h>").  The
first three @casts don't have the header file parameter, so only
look in the vmlinux binary (if found).

My guess is that if the first three @cast's added a "kernel<BAR.h>",
it'd work even on your box, without kernel debuginfo.  One just
needs the right BAR.h.

And a good drink at said BAR.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug tapsets/19300] Many SystemTap tapsets failing on kernel.org 4.2.0 kernel

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19300

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |WORKSFORME

--- Comment #5 from Frank Ch. Eigler <fche at redhat dot com> ---
no recent reports, newer kernels OK too

-- 
You are receiving this mail because:
You are the assignee for the bug.