SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Aleksander Alekseev]

Hello

Today I decided to try SystemTap 3.0. Here is exactly what I did.

For my experiments I created a new KVM virtual machine and installed a
clean Ubuntu 14.04.4 system from ISO image named:

ubuntu-14.04.4-server-amd64.iso

This distribution uses Linux kernel 4.2.0.

At first I tried to use default systemtap package:

sudo apt-get update
sudo apt-get install systemtap gcc
sudo apt-get install linux-headers-$(uname -r)

Also I downloaded and installed:

linux-image-4.2.0-27-generic-dbgsym_4.2.0-27.32_amd64.ddeb

... from here http://ddebs.ubuntu.com/pool/main/l/linux/

Unfortunately SystemTap didn't work:

http://pastebin.com/raw/GDry4JWJ

According to copyright message it looks quite old so I decided to try
brand new SystemTap 3.0:

```
sudo apt-get remove systemtap
sudo apt-get install m4 g++ gettext libz-dev
wget https://sourceware.org/systemtap/ftp/releases/systemtap-3.0.tar.gz
wget \
  https://fedorahosted.org/releases/e/l/elfutils/0.159/elfutils-0.159.tar.bz2
tar -xvzf systemtap-3.0.tar.gz
tar -xvjf elfutils-0.159.tar.bz2
cd systemtap-3.0
./configure --with-elfutils=../elfutils-0.159
make -j2
sudo make install
```

As a side node SystemTap 3.0 doesn't compile with newer elfutils 0.165.

It turned out that newer SystemTap doesn't work on this system either:

http://pastebin.com/raw/HvxaxFYw

I found description of these error messages here:


"""
Error -84 is EILSEQ: "illegal byte sequence". The kernel can return
this signal, for example if it cannot decode its own binary in the
region nearby the requested kprobe, or it finds instruction patterns
that its decoder has not been taught to handle. It's not a systemtap
bug.
"""

-- 
Best regards,
Aleksander Alekseev
http://eax.me/

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Aleksander Alekseev]

> Hello
> 
> Today I decided to try SystemTap 3.0. Here is exactly what I did.
> 
> For my experiments I created a new KVM virtual machine and installed a
> clean Ubuntu 14.04.4 system from ISO image named:
> 
> ubuntu-14.04.4-server-amd64.iso
> 
> This distribution uses Linux kernel 4.2.0.
> 
> At first I tried to use default systemtap package:
> 
> sudo apt-get update
> sudo apt-get install systemtap gcc
> sudo apt-get install linux-headers-$(uname -r)
> 
> Also I downloaded and installed:
> 
> linux-image-4.2.0-27-generic-dbgsym_4.2.0-27.32_amd64.ddeb
> 
> ... from here http://ddebs.ubuntu.com/pool/main/l/linux/
> 
> Unfortunately SystemTap didn't work:
> 
> http://pastebin.com/raw/GDry4JWJ
> 
> According to copyright message it looks quite old so I decided to try
> brand new SystemTap 3.0:
> 
> ```
> sudo apt-get remove systemtap
> sudo apt-get install m4 g++ gettext libz-dev
> wget
> https://sourceware.org/systemtap/ftp/releases/systemtap-3.0.tar.gz
> wget \
> https://fedorahosted.org/releases/e/l/elfutils/0.159/elfutils-0.159.tar.bz2
> tar -xvzf systemtap-3.0.tar.gz tar -xvjf elfutils-0.159.tar.bz2
> cd systemtap-3.0
> ./configure --with-elfutils=../elfutils-0.159
> make -j2
> sudo make install
> ```
> 
> As a side node SystemTap 3.0 doesn't compile with newer elfutils
> 0.165.
> 
> It turned out that newer SystemTap doesn't work on this system either:
> 
> http://pastebin.com/raw/HvxaxFYw
> 
> I found description of these error messages here:
> 
> 
> """
> Error -84 is EILSEQ: "illegal byte sequence". The kernel can return
> this signal, for example if it cannot decode its own binary in the
> region nearby the requested kprobe, or it finds instruction patterns
> that its decoder has not been taught to handle. It's not a systemtap
> bug.
> """
> 

Oops, my email client accidentally sent this message before I finished
it. "Here" supposed to be http://stackoverflow.com/a/22436660/1565238

Currently I have no idea what to do next so any advices would be
appreciated. Naturally I would be happy to provide any debug
information you might need.

-- 
Best regards,
Aleksander Alekseev
http://eax.me/

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Mark Wielaard]

On Tue, 2016-03-29 at 16:30 +0300, Aleksander Alekseev wrote:
> As a side node SystemTap 3.0 doesn't compile with newer elfutils
> 0.165.

What issues do you have?
It is certainly supposed to build against elfutils 0.165.

Thanks,

Mark

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Josh Stone]

On 03/29/2016 06:34 AM, Aleksander Alekseev wrote:
>> Hello
>>
>> Today I decided to try SystemTap 3.0. Here is exactly what I did.
>>
>> For my experiments I created a new KVM virtual machine and installed a
>> clean Ubuntu 14.04.4 system from ISO image named:
>>
>> ubuntu-14.04.4-server-amd64.iso
>>
>> This distribution uses Linux kernel 4.2.0.
>>
>> At first I tried to use default systemtap package:
>>
>> sudo apt-get update
>> sudo apt-get install systemtap gcc
>> sudo apt-get install linux-headers-$(uname -r)
>>
>> Also I downloaded and installed:
>>
>> linux-image-4.2.0-27-generic-dbgsym_4.2.0-27.32_amd64.ddeb
>>
>> ... from here http://ddebs.ubuntu.com/pool/main/l/linux/
>>
>> Unfortunately SystemTap didn't work:
>>
>> http://pastebin.com/raw/GDry4JWJ
>>
>> According to copyright message it looks quite old so I decided to try
>> brand new SystemTap 3.0:
>>
>> ```
>> sudo apt-get remove systemtap
>> sudo apt-get install m4 g++ gettext libz-dev
>> wget
>> https://sourceware.org/systemtap/ftp/releases/systemtap-3.0.tar.gz
>> wget \
>> https://fedorahosted.org/releases/e/l/elfutils/0.159/elfutils-0.159.tar.bz2
>> tar -xvzf systemtap-3.0.tar.gz tar -xvjf elfutils-0.159.tar.bz2
>> cd systemtap-3.0
>> ./configure --with-elfutils=../elfutils-0.159
>> make -j2
>> sudo make install
>> ```
>>
>> As a side node SystemTap 3.0 doesn't compile with newer elfutils
>> 0.165.
>>
>> It turned out that newer SystemTap doesn't work on this system either:
>>
>> http://pastebin.com/raw/HvxaxFYw
>>
>> I found description of these error messages here:
>>
>>
>> """
>> Error -84 is EILSEQ: "illegal byte sequence". The kernel can return
>> this signal, for example if it cannot decode its own binary in the
>> region nearby the requested kprobe, or it finds instruction patterns
>> that its decoder has not been taught to handle. It's not a systemtap
>> bug.
>> """
>>
> 
> Oops, my email client accidentally sent this message before I finished
> it. "Here" supposed to be http://stackoverflow.com/a/22436660/1565238

As noted in that answer, can you provide "objdump -d" around those
addresses?

Are you sure the dbgsym you downloaded is an exact match for your
kernel?  It should have failed earlier if it wasn't, but a mismatch
could lead us to attempt kprobes on weird addresses, like in the middle
of an instruction.

Another thing to try is instead use "nd_syscall.open" etc. which will
probe functions by name, without requiring debuginfo at all.

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Aleksander Alekseev]

Hello

Thanks for your replies!

> What issues do you have?
> It is certainly supposed to build against elfutils 0.165.

`make` fails like this:

http://pastebin.com/raw/afHGbFpE

> As noted in that answer, can you provide "objdump -d" around those
> addresses?

Could you please explain how to find reported address (e.g.
0xffffffff81205870) in objdump output? Here is a full output of `sudo
objdump -dr /boot/vmlinuz-4.2.0-27-generic` command:

https://www.dropbox.com/s/w9iywua0c6pdkju/vmlinux.decoded.gz?dl=0

I hope at list this is a right assembly code? :)

Also here is an output of `stap -vv`:

http://pastebin.com/raw/GEdMqL7U

> Are you sure the dbgsym you downloaded is an exact match for your
> kernel?  It should have failed earlier if it wasn't, but a mismatch
> could lead us to attempt kprobes on weird addresses, like in the
> middle of an instruction.

`uname -r` says `4.2.0-27-generic` and installed packages are:

linux-headers-4.2.0-27
linux-headers-4.2.0-27-generic
linux-image-4.2.0-27-generic
linux-image-4.2.0-27-generic-dbgsym
linux-image-extra-4.2.0-27-generic

Looks right. Is it any other way to check this?

> Another thing to try is instead use "nd_syscall.open" etc. which will
> probe functions by name, without requiring debuginfo at all.

Wow, it works! So... why I need debuginfo in the first place? Are there
any features that will not work without it or something like this?

-- 
Best regards,
Aleksander Alekseev
http://eax.me/

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Frank Ch. Eigler]

Aleksander Alekseev wrote:

> [...]
> ubuntu-14.04.4-server-amd64.iso
> This distribution uses Linux kernel 4.2.0.
> [...]
> linux-image-4.2.0-27-generic-dbgsym_4.2.0-27.32_amd64.ddeb
> Unfortunately SystemTap didn't work:
> http://pastebin.com/raw/GDry4JWJ

Strange.  I have what seems like an identical ubuntu VM ...:

% dpkg -l | egrep '(systemtap.*2.3)|(linux.*4.2)'
ii  linux-headers-4.2.0-27               4.2.0-27.32~14.04.1                     all          Header files related to Linux kernel version 4.2.0
ii  linux-headers-4.2.0-27-generic       4.2.0-27.32~14.04.1                     amd64        Linux kernel headers for version 4.2.0 on 64 bit x86 SMP
ii  linux-image-4.2.0-27-generic         4.2.0-27.32~14.04.1                     amd64        Linux kernel image for version 4.2.0 on 64 bit x86 SMP
ii  linux-image-4.2.0-27-generic-dbgsym  4.2.0-27.32~14.04.1                     amd64        Linux kernel debug image for version 4.2.0 on 64 bit x86 SMP
ii  systemtap                            2.3-1ubuntu1.4                          amd64        instrumentation system for Linux
ii  systemtap-common                     2.3-1ubuntu1.4                          all          instrumentation system for Linux (common component)
ii  systemtap-runtime                    2.3-1ubuntu1.4                          amd64        instrumentation system for Linux (runtime component)

and stap things just work (tm):
% sudo stap exec.stp
systemd-udevd(3480) exit (0)
^Cstapio(3479) exit (0)

% stap -p2 exec.stp | grep kernel.function

kernel.function("SyS_execve@/build/linux-lts-wily-Vv6Eyd/linux-lts-wily-4.2.0/fs/exec.c:1708").call /* pc=_stext+0x1f2cd8 */ /* <- syscall.execve = kernel.function("sys_execve").call <- syscall.execve */
kernel.function("do_exit@/build/linux-lts-wily-Vv6Eyd/linux-lts-wily-4.2.0/kernel/exit.c:653").call /* pc=_stext+0x79818 */ /* <- syscall.exit = kernel.function("do_exit").call <- syscall.exit */


- FChE

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Josh Stone]

On 03/29/2016 02:53 PM, Aleksander Alekseev wrote:
> Hello
> 
> Thanks for your replies!
> 
>> What issues do you have?
>> It is certainly supposed to build against elfutils 0.165.
> 
> `make` fails like this:
> 
> http://pastebin.com/raw/afHGbFpE
> 
>> As noted in that answer, can you provide "objdump -d" around those
>> addresses?
> 
> Could you please explain how to find reported address (e.g.
> 0xffffffff81205870) in objdump output? Here is a full output of `sudo
> objdump -dr /boot/vmlinuz-4.2.0-27-generic` command:
> 
> https://www.dropbox.com/s/w9iywua0c6pdkju/vmlinux.decoded.gz?dl=0
> 
> I hope at list this is a right assembly code? :)

Hmm, right, I haven't actually tried that in a while.  objdump doesn't
make it easy to output a specific function, but there's --start-address
and --stop-address if you can figure out the right offsets.

I think objdump does try to decompress vmlinuz, but the results have a
lot of "(bad)" lines, so I don't really trust this.  You should use the
kernel's script to extract it first:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/scripts/extract-vmlinux

> Also here is an output of `stap -vv`:
> 
> http://pastebin.com/raw/GEdMqL7U

OK, here we see pc=0xffffffff811fca70.  I downloaded your dbgsym
4.2.0-27.32 myself, and objdump gives me:

ffffffff811fca70 <SyS_open>:
ffffffff811fca70:       e8 cb 7a 5f 00          callq  ffffffff817f4540
<__fentry__>

This looks perfect.

But the closest kernel I could find is 4.2.0-27.32~14.04.1, and this
doesn't match at all:

ffffffff811fca6b:       48 c7 83 80 00 00 00    movq   $0x0,0x80(%rbx)
ffffffff811fca72:       00 00 00 00
ffffffff811fca76:       44 89 b3 88 00 00 00    mov    %r14d,0x88(%rbx)

ca70 would be probing in the middle of an instruction!

At least "readelf -n" on these two does show different Build ID values.

>> Are you sure the dbgsym you downloaded is an exact match for your
>> kernel?  It should have failed earlier if it wasn't, but a mismatch
>> could lead us to attempt kprobes on weird addresses, like in the
>> middle of an instruction.
> 
> `uname -r` says `4.2.0-27-generic` and installed packages are:
> 
> linux-headers-4.2.0-27
> linux-headers-4.2.0-27-generic
> linux-image-4.2.0-27-generic
> linux-image-4.2.0-27-generic-dbgsym
> linux-image-extra-4.2.0-27-generic
> 
> Looks right. Is it any other way to check this?

Try: dpkg --list 'linux-image*'

Ubuntu sometimes has sub-versions that don't show up in uname, like .32
in your dbgsym vs the .32~14.04.1 security update I found.  The running
kernel must match dbgsym exactly for us to do anything useful with it,
but like I mentioned we do runtime build-id checks which should catch
that before kprobes are attempted.

If that mismatch isn't being detected, that's a bug in itself!

I can't find exact dbgsym for the linux-image I downloaded.  If Ubuntu
doesn't publish this, you'll have to complain to them, or build and run
your own kernel with the necessary debuginfo.

>> Another thing to try is instead use "nd_syscall.open" etc. which will
>> probe functions by name, without requiring debuginfo at all.
> 
> Wow, it works! So... why I need debuginfo in the first place? Are there
> any features that will not work without it or something like this?

Without debuginfo, we can only use kprobes on the call and return of a
function by name; no probing inline functions or specific line numbers.
 We also don't get location info for any variables.  The nd_syscall
tapset has hard-coded the syscall parameter positions and reads them
according to normal calling ABI, but you can't rely on that everywhere.

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Josh Stone]

On 03/29/2016 04:02 PM, Frank Ch. Eigler wrote:
> 
> Aleksander Alekseev wrote:
> 
>> [...]
>> ubuntu-14.04.4-server-amd64.iso
>> This distribution uses Linux kernel 4.2.0.
>> [...]
>> linux-image-4.2.0-27-generic-dbgsym_4.2.0-27.32_amd64.ddeb
>> Unfortunately SystemTap didn't work:
>> http://pastebin.com/raw/GDry4JWJ
> 
> Strange.  I have what seems like an identical ubuntu VM ...:
> 
> % dpkg -l | egrep '(systemtap.*2.3)|(linux.*4.2)'
> ii  linux-headers-4.2.0-27               4.2.0-27.32~14.04.1                     all          Header files related to Linux kernel version 4.2.0
> ii  linux-headers-4.2.0-27-generic       4.2.0-27.32~14.04.1                     amd64        Linux kernel headers for version 4.2.0 on 64 bit x86 SMP
> ii  linux-image-4.2.0-27-generic         4.2.0-27.32~14.04.1                     amd64        Linux kernel image for version 4.2.0 on 64 bit x86 SMP
> ii  linux-image-4.2.0-27-generic-dbgsym  4.2.0-27.32~14.04.1                     amd64        Linux kernel debug image for version 4.2.0 on 64 bit x86 SMP

Huh, the file on ddebs.ubuntu.com doesn't have "~14.04.1".  I wonder
where yours can be downloaded?

> ii  systemtap                            2.3-1ubuntu1.4                          amd64        instrumentation system for Linux
> ii  systemtap-common                     2.3-1ubuntu1.4                          all          instrumentation system for Linux (common component)
> ii  systemtap-runtime                    2.3-1ubuntu1.4                          amd64        instrumentation system for Linux (runtime component)
> 
> and stap things just work (tm):
> % sudo stap exec.stp
> systemd-udevd(3480) exit (0)
> ^Cstapio(3479) exit (0)
> 
> % stap -p2 exec.stp | grep kernel.function
> 
> kernel.function("SyS_execve@/build/linux-lts-wily-Vv6Eyd/linux-lts-wily-4.2.0/fs/exec.c:1708").call /* pc=_stext+0x1f2cd8 */ /* <- syscall.execve = kernel.function("sys_execve").call <- syscall.execve */
> kernel.function("do_exit@/build/linux-lts-wily-Vv6Eyd/linux-lts-wily-4.2.0/kernel/exit.c:653").call /* pc=_stext+0x79818 */ /* <- syscall.exit = kernel.function("do_exit").call <- syscall.exit */
> 
> 
> - FChE
> 

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Frank Ch. Eigler]

jistone wrote:

>> % dpkg -l | egrep '(systemtap.*2.3)|(linux.*4.2)'
>> ii  linux-headers-4.2.0-27               4.2.0-27.32~14.04.1                     all          Header files related to Linux kernel version 4.2.0
>> [...]
>
> Huh, the file on ddebs.ubuntu.com doesn't have "~14.04.1".  I wonder
> where yours can be downloaded?

Dunno - my /etc/apt/sources.list.d/ddebs.list has this:

deb http://ddebs.ubuntu.com trusty main restricted universe multiverse
deb http://ddebs.ubuntu.com trusty-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com trusty-security main restricted universe multiverse
deb http://ddebs.ubuntu.com trusty-proposed main restricted universe multiverse

and all the kernel files etc. just came from routine apt-get update/upgrade.

% apt-cache showpkg linux-headers-4.2.0-27
Package: linux-headers-4.2.0-27
Versions: 
4.2.0-27.32~14.04.1 (/var/lib/apt/lists/ca.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-amd64_Packages) (/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages) (/var/lib/dpkg/status)
[...]

Just browsing around their http servers' directory, that seems to come
out of the linux-lts-wily hierarchy, where that ~14.04.1 substring
does appear in the file names.

- FChE

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Mark Wielaard]

On Wed, 2016-03-30 at 00:53 +0300, Aleksander Alekseev wrote:
> Hello
> 
> Thanks for your replies!
> 
> > What issues do you have?
> > It is certainly supposed to build against elfutils 0.165.
> 
> `make` fails like this:
> 
> http://pastebin.com/raw/afHGbFpE
>
> In file included from include-elfutils/gelf.h:32:0,
>                  from include-elfutils/elfutils/libdw.h:32,
>                  from session.h:29,
>                  from tapset-dynprobe.cxx:12:
> include-elfutils/libelf.h:280:8: error: ‘Elf32_Chdr’ does not name a type
>  extern Elf32_Chdr *elf32_getchdr (Elf_Scn *__scn);
>         ^

That is because your glibc is old and elf.h doesn't contain the new
definitions. That has been fixed by:
https://git.fedorahosted.org/cgit/elfutils.git/commit/?id=344ca0775da729e1bfdd61bb88ba4c64befece07
Which should also be included in the debian/ubuntu package sources.

It should also be in 0.166 to be released later this week.

Cheers,

Mark

Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Aleksander Alekseev]

Hello

> That is because your glibc is old and elf.h doesn't contain the new
> definitions.

Oh, I see.

> Dunno - my /etc/apt/sources.list.d/ddebs.list has this:
> [...]
> Just browsing around their http servers' directory, that seems to come
> out of the linux-lts-wily hierarchy, where that ~14.04.1 substring
> does appear in the file names.

That helped! I installed ...-dbgsym package version 4.2.0-27.32
~14.04.1 using your ddebs.list and everything works just fine now.

Thanks a lot for help everyone!

-- 
Best regards,
Aleksander Alekseev
http://eax.me/