[Bug server/20541] New: Systemtap 3.0 installation problem on Ubuntu 16.04 LTS @GCP

[Bug server/20541] New: Systemtap 3.0 installation problem on Ubuntu 16.04 LTS @GCP

[steve.tsai at nexusguard dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20541

            Bug ID: 20541
           Summary: Systemtap 3.0 installation problem on Ubuntu 16.04 LTS
                    @GCP
           Product: systemtap
           Version: unspecified
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: server
          Assignee: systemtap at sourceware dot org
          Reporter: steve.tsai at nexusguard dot com
  Target Milestone: ---

Dear Systemtap Experts:

If this is not the place to post this kind of issue, please let me know.
I have successfully installed systamtap on Ubuntu 14.04 LTS.
However, I have a problem to try to install systemtap on Ubuntu 16.04 LTS. I
started with a vm instance from Google cloud platform.

I successfully installed systemtap both from 
apt-get install systeamtap 
and 
ElfUtils 0.167 & SystemTap 3.0 from Source Packages.

But I got the same error message.

$ bash
$ sudo stap -e 'probe begin { printf("Hello, World!\n"); exit() }'
Hello, World!
$ sudo stap -ve 'probe kernel.function("sys_open") {log("hello world") exit()}'
Pass 1: parsed user script and 110 library script(s) using
108356virt/43892res/6300shr/37664data kb, in 140usr/20sys/154rea
l ms.
Pass 2: analyzed script: 1 probe(s), 2 function(s), 0 embed(s), 0 global(s)
using 154732virt/91192res/7236shr/84040data kb,
 in 870usr/380sys/5302real ms.
Pass 3: translated to C into
"/tmp/stap1nFy6I/stap_382ffff0246776764576b305cb895aef_1314_src.c" using
154732virt/91384res/7
428shr/84040data kb, in 10usr/60sys/78real ms.
Pass 4: compiled C into "stap_382ffff0246776764576b305cb895aef_1314.ko" in
1530usr/240sys/2122real ms.
Pass 5: starting run.
WARNING: probe
kernel.function("SyS_open@/build/linux-lts-xenial-QiVniY/linux-lts-xenial-4.4.0/fs/open.c:1038")
(address 0x
ffffffff811fcd40) registration error (rc -84)

------------------------------------------------------
$ uname -a
Linux steve-systemtap2 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux

---------------------------------------------
My complete installation process for case 1.
sudo apt-get install -y systemtap gcc
sudo apt-get install -y linux-headers-generic

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
ECDCAD72428D7C01

sudo vi /etc/apt/sources.list.d/ddebs.list

deb http://ddebs.ubuntu.com/ trusty main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-security main restricted universe
multiverse
deb http://ddebs.ubuntu.com/ trusty-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-proposed main restricted universe
multiverse

sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym

#bash           // to make $PATH work
#sudo stap -e 'probe begin { printf("Hello, World!\n"); exit() }'
#sudo stap -ve 'probe kernel.function("sys_open") {log("hello world") exit()}'
------------------------------------------------------------
My complete installation process for case 2.

sudo apt-get install -y zlib1g-dev libdw-dev g++ gcc make m4 gettext libcap-dev
linux-headers-$(uname -r)

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
ECDCAD72428D7C01

sudo vi /etc/apt/sources.list.d/ddebs.list

deb http://ddebs.ubuntu.com/ trusty main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-security main restricted universe
multiverse
deb http://ddebs.ubuntu.com/ trusty-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ trusty-proposed main restricted universe
multiverse

sudo apt-get update
sudo apt-get install linux-image-$(uname -r)-dbgsym

mkdir systemtap
cd systemtap

wget
https://fedorahosted.org/releases/e/l/elfutils/0.167/elfutils-0.167.tar.bz2
tar jxf elfutils-0.167.tar.bz2

wget https://sourceware.org/systemtap/ftp/releases/systemtap-3.0.tar.gz
tar zxf systemtap-3.0.tar.gz
cd systemtap-3.0
sudo -i
./configure --with-elfutils=../elfutils-0.167
make
sudo make install
-----------------------------------------------------------------------

Did I make installation wrong? or Is there a bug?
Could you help me to solve this problem?

thank you in advance.

Steve

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug server/20541] Systemtap 3.0 installation problem on Ubuntu 16.04 LTS @GCP

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20541

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
Your build appears fine, but:

> Pass 5: starting run.
> WARNING: probe
> kernel.function("SyS_open@/build/linux-lts-xenial-QiVniY/linux-lts-xenial-4.
> 4.0/fs/open.c:1038") (address 0x
> ffffffff811fcd40) registration error (rc -84)

This error is coming from the kernel, as it rejects our proposed probe.
Does

# sudo grep -i sys.open /proc/kallsyms

give a matching address (...811fcd40) on your machine?  If so, stap probably
isn't doing anything wrong, and we'll have to work around kernel limitations.

One possibility is to use

# stap -P ...

(prologue-searching to offset probe PC).  Another is to smoke-test stap against
other probe points

# stap -e 'probe nd_syscall.* { }' -c 'sleep 10'

to see whether many other functions are affected by this registration error,
only only this one.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug server/20541] Systemtap 3.0 installation problem on Ubuntu 16.04 LTS @GCP

[steve.tsai at nexusguard dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20541

--- Comment #2 from SteveTsai <steve.tsai at nexusguard dot com> ---
Thank you, Frank.
The followings are some test results that you suggested.
-----------------------------------------------------
$ sudo stap -P -ve 'probe kernel.function("sys_open") {log("hello world")
exit()}'
Pass 5: starting run.
WARNING: probe
kernel.function("SyS_open@/build/linux-lts-xenial-QiVniY/linux-lts-xenial-4.4.0/fs/open.c:1038")
(address 0xff
ffffff811fcd46) registration error (rc -84)

$ sudo grep -i sys.open /proc/kallsyms
ffffffff81209ba0 t perf_trace_do_sys_open
ffffffff81209d20 t trace_event_raw_event_do_sys_open
ffffffff81209e50 t trace_raw_output_do_sys_open
ffffffff8120c120 T do_sys_open
ffffffff8120c3c0 T SyS_open
ffffffff8120c3c0 T sys_open
ffffffff8120c3e0 T SyS_openat
ffffffff8120c3e0 T sys_openat
ffffffff81265bf0 T compat_SyS_open
ffffffff81265bf0 T compat_sys_open
ffffffff81265c10 T compat_SyS_openat
ffffffff81265c10 T compat_sys_openat
ffffffff81266500 T compat_SyS_open_by_handle_at
ffffffff81266500 T compat_sys_open_by_handle_at
ffffffff81271040 T SyS_open_by_handle_at
ffffffff81271040 T sys_open_by_handle_at
ffffffff812856d0 t proc_sys_open
ffffffff81d6cbd8 r __tracepoint_ptr_do_sys_open
ffffffff81d6ea50 r __tpstrtab_do_sys_open
ffffffff81e76de0 d print_fmt_do_sys_open
ffffffff81e76e40 d trace_event_type_funcs_do_sys_open
ffffffff81e76f00 d event_do_sys_open
ffffffff81f05a60 d event_class_do_sys_open
ffffffff81f11260 D __tracepoint_do_sys_open
ffffffff81f8fcaa t trace_event_define_fields_do_sys_open
ffffffff82088200 t __event_do_sys_open

$ sudo stap -e 'probe nd_syscall.* { }' -c 'sleep 10'
WARNING: side-effect-free probe 'probe_2791': identifier 'nd_syscall' at
<input>:1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_2818': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_2826': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_2878': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_2880': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_2934': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3025': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3076': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3077': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3079': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3080': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3082': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3083': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3101': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3103': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3105': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3142': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3147': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3148': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3159': identifier '__nd_syscall' at
/usr/share/systemtap/tapset/linux/nd_syscalls.stp:
2928:33
 source:                                 __nd_syscall.inotify_init ?
                                         ^
WARNING: side-effect-free probe 'probe_3210': identifier 'nd_syscall' at
<input>:1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3319': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3327': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3340': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3353': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3357': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3377': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3435': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3437': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3445': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3447': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3464': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3465': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3504': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3609': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3627': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3657': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3663': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3701': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3705': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3709': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3715': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3716': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3807': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
WARNING: side-effect-free probe 'probe_3809': identifier 'nd_syscall' at :1:7
 source: probe nd_syscall.* { }
               ^
--------------------------------------------------------
Any idea?

Thank you very much.
Steve

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug server/20541] Systemtap 3.0 installation problem on Ubuntu 16.04 LTS @GCP

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20541

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
   Last reconfirmed|                            |2017-05-09
     Ever confirmed|0                           |1

--- Comment #3 from Frank Ch. Eigler <fche at redhat dot com> ---
Sorry for the delay.
It sounds as though the debuginfo-based probes (kernel.function) are getting
wrong addresses for your kernel, whereas the symbol-table-based ones
(nd_syscall*) are working.  I suspect a mismatch between your running kernel
and the exact dbgsym version installed.

The % stap-report 
program gathers information that may help further diagnosis.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug server/20541] Systemtap 3.0 installation problem on Ubuntu 16.04 LTS @GCP

[fche at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=20541

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|---                         |OBSOLETE

--- Comment #4 from Frank Ch. Eigler <fche at redhat dot com> ---
likely an installation problem, eased by debuginfod deployment for ubuntu and
elsewhere

-- 
You are receiving this mail because:
You are the assignee for the bug.