[Bug translator/16914] New: probe syscall.* fails with compilation error

[Bug translator/16914] New: probe syscall.* fails with compilation error

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

            Bug ID: 16914
           Summary: probe syscall.* fails with compilation error
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: translator
          Assignee: systemtap at sourceware dot org
          Reporter: mcermak at redhat dot com

Created attachment 7588
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7588&action=edit
stap output

On aarch64, stap -e 'probe syscall.* { log("hey!"); exit() }' returns errors
like this:

/tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c: In function
‘probe_2344’:
/tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c:11776:43:
error: ‘__NR_compat_clone’ undeclared (first use in this function)
       if ((((l->l___nr) != ((((int64_t) ( __NR_compat_clone ))))))) {
                                           ^
/tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c:11776:43:
note: each undeclared identifier is reported only once for each function it
appears in
/tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c: In function
‘probe_2346’:
/tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c:11907:43:
error: ‘__NR_compat_close’ undeclared (first use in this function)
       if ((((l->l___nr) != ((((int64_t) ( __NR_compat_close ))))))) {
                                           ^
/tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c: In function
‘probe_2356’:
/tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c:12038:43:
error: ‘__NR_compat_futimesat’ undeclared (first use in this function)
       if ((((l->l___nr) != ((((int64_t) ( __NR_compat_futimesat ))))))) {

[ ... ]

 # rpm -q kernel systemtap
kernel-3.13.0-0.rc7.31.sa2.bz1094366.aarch64 (3.13.0-0.rc7.31 + rhbz1094366
fix)
systemtap-2.6-1.sa1.3.aarch64 (fresh upstream master head)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #1 from Martin Cermak <mcermak at redhat dot com> ---
Created attachment 7589
  --> https://sourceware.org/bugzilla/attachment.cgi?id=7589&action=edit
module source

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #2 from David Smith <dsmith at redhat dot com> ---
(In reply to Martin Cermak from comment #0)
> Created attachment 7588 [details]
> stap output
> 
> On aarch64, stap -e 'probe syscall.* { log("hey!"); exit() }' returns errors
> like this:
> 
> /tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c: In
> function ‘probe_2344’:
> /tmp/stapV76Au7/stap_19d2b02d95ec8b13d55cf356928e49ab_143230_src.c:11776:43:
> error: ‘__NR_compat_clone’ undeclared (first use in this function)
>        if ((((l->l___nr) != ((((int64_t) ( __NR_compat_clone ))))))) {

I understand what is going on here, but I need some aarch64 education to know
how to fix it. Does aarch64 run 32-bit executables? Is the CONFIG_COMPAT option
on on an aarch64 kernel?

If so, aarch64 will need its own section in runtime/linux/compat_unistd.h.

If not, something wacky is going on and that code should have never been hit.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #3 from Martin Cermak <mcermak at redhat dot com> ---
(In reply to David Smith from comment #2)
> Does aarch64 run 32-bit executables?
My *guess* is that it doesn't. Certainly gcc doesn't support -m32.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[jistone at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com

--- Comment #4 from Josh Stone <jistone at redhat dot com> ---
(In reply to Martin Cermak from comment #0)
> kernel-3.13.0-0.rc7.31.sa2.bz1094366.aarch64 (3.13.0-0.rc7.31 + rhbz1094366 fix)

This kernel does have CONFIG_COMPAT=y, but I don't think that's typical for
that product.  Regardless, we need to be prepared for the possibility.

(In reply to Martin Cermak from comment #3)
> (In reply to David Smith from comment #2)
> > Does aarch64 run 32-bit executables?
> My *guess* is that it doesn't. Certainly gcc doesn't support -m32.

The architecture does support 32-bit, but distros might choose not to enable it
in the kernel or toolchain.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #5 from David Smith <dsmith at redhat dot com> ---
(In reply to Josh Stone from comment #4)
> (In reply to Martin Cermak from comment #0)
> > kernel-3.13.0-0.rc7.31.sa2.bz1094366.aarch64 (3.13.0-0.rc7.31 + rhbz1094366 fix)
> 
> This kernel does have CONFIG_COMPAT=y, but I don't think that's typical for
> that product.  Regardless, we need to be prepared for the possibility.

Here's the description for CONFIG_COMPAT from the kernel's arm/arm64/Kconfig
file:

====
      This option enables support for a 32-bit EL0 running under a 64-bit
      kernel at EL1. AArch32-specific components such as system calls,
      the user helper functions, VFP support and the ptrace interface are
      handled appropriately by the kernel.

      If you want to execute 32-bit userspace applications, say Y.
====

So right now the kernel supports 32-bit apps, but the toolchain doesn't. Odd,
but OK.

From poking around the kernel source, it appears that on arm64, syscall numbers
for 32-bit syscalls can differ from their 64-bit versions, but most are the
same.

I've checked in commit 48ad5c5 which should hopefully fix this.

Martin or Josh, I'll need one of you to test this and let me know if it worked.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #6 from Martin Cermak <mcermak at redhat dot com> ---
Hi David, now, with current upstream head, I'm getting: 

--------------------8<----------------------------------------------

S aarch64 # stap -e 'probe syscall.* { log("hey!"); exit() }'
In file included from
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c:28:0:
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c: In function
‘probe_2357’:
/usr/share/systemtap/runtime/linux/compat_unistd.h:116:32: error:
‘__NR_futimesat’ undeclared (first use in this function)
 #define __NR_compat_futimesat  __NR_futimesat
                                ^
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c:12038:43:
note: in expansion of macro ‘__NR_compat_futimesat’
       if ((((l->l___nr) != ((((int64_t) ( __NR_compat_futimesat ))))))) {
                                           ^
/usr/share/systemtap/runtime/linux/compat_unistd.h:116:32: note: each
undeclared identifier is reported only once for each function it appears in
 #define __NR_compat_futimesat  __NR_futimesat
                                ^
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c:12038:43:
note: in expansion of macro ‘__NR_compat_futimesat’
       if ((((l->l___nr) != ((((int64_t) ( __NR_compat_futimesat ))))))) {
                                           ^
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c: In function
‘probe_2496’:
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c:13332:33:
error: ‘__NR_futimesat’ undeclared (first use in this function)
           })) != ((((int64_t) ( __NR_futimesat ))))))) {
                                 ^
In file included from
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c:28:0:
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c: In function
‘probe_2700’:
/usr/share/systemtap/runtime/linux/compat_unistd.h:121:27: error: ‘__NR_open’
undeclared (first use in this function)
 #define __NR_compat_open  __NR_open
                           ^
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c:14433:43:
note: in expansion of macro ‘__NR_compat_open’
       if ((((l->l___nr) != ((((int64_t) ( __NR_compat_open ))))))) {
                                           ^
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c: At top
level:
/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.c:17973:2:
error: #error "Need CONFIG_KPROBES!"
 #error "Need CONFIG_KPROBES!"
  ^
make[1]: ***
[/tmp/stapnP5rLb/stap_1c56b4d79e886a846bcb2bcb2c11d0e9_143230_src.o] Error 1
make: *** [_module_/tmp/stapnP5rLb] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compilation failed.  [man error::pass4]
S aarch64 #

--------------------8<----------------------------------------------

which, although apparently we're not yet there, looks much better!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #7 from David Smith <dsmith at redhat dot com> ---
I found an aarch64 machine I could access
(running3.13.0-0.rc7.31.sa2.k32v1.aarch64.debug), built, installed, and got the
following:

====
# stap -p4 -e 'probe syscall.* { log("hey!"); exit() }'
In file included from
/usr/src/kernels/3.13.0-0.rc7.31.sa2.k32v1.aarch64.debug/arch/arm64/include/asm/dma-mapping.h:27:0,
                 from include/linux/dma-mapping.h:76,
                 from include/linux/skbuff.h:33,
                 from include/linux/netlink.h:6,
                 from
/tmp/stapSX3Lm0/stap_a9d73a0f48c94b332f8efe28f9c0739d_104773_src.c:127:
/usr/src/kernels/3.13.0-0.rc7.31.sa2.k32v1.aarch64.debug/arch/arm64/include/asm/xen/hypervisor.h:1:50:
fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
                                                  ^
compilation terminated.
make[1]: ***
[/tmp/stapSX3Lm0/stap_a9d73a0f48c94b332f8efe28f9c0739d_104773_src.o] Error 1
make: *** [_module_/tmp/stapSX3Lm0] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compilation failed.  [man error::pass4]
====

So that looks like a kernel problem, not a systemtap problem.

On your kernel, something odd is going on if things like __NR_open aren't
defined. Can you try the following patch and see if that fixes things?

====
diff --git a/runtime/linux/compat_unistd.h b/runtime/linux/compat_unistd.h
index 25ca1bd..11abc26 100644
--- a/runtime/linux/compat_unistd.h
+++ b/runtime/linux/compat_unistd.h
@@ -11,6 +11,8 @@
 #ifndef _COMPAT_UNISTD_H_
 #define _COMPAT_UNISTD_H_

+#include <linux/unistd.h>
+
 #if defined(__x86_64__)

 // On older kernels (like RHEL5), we have to define our own 32-bit
====

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[mcermak at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #8 from Martin Cermak <mcermak at redhat dot com> ---
(In reply to David Smith from comment #7)
> I found an aarch64 machine I could access
> (running3.13.0-0.rc7.31.sa2.k32v1.aarch64.debug), built, installed, and got
> the following:
> 
> ====
> # stap -p4 -e 'probe syscall.* { log("hey!"); exit() }'
> In file included from
> /usr/src/kernels/3.13.0-0.rc7.31.sa2.k32v1.aarch64.debug/arch/arm64/include/
> asm/dma-mapping.h:27:0,
>                  from include/linux/dma-mapping.h:76,
>                  from include/linux/skbuff.h:33,
>                  from include/linux/netlink.h:6,
>                  from
> /tmp/stapSX3Lm0/stap_a9d73a0f48c94b332f8efe28f9c0739d_104773_src.c:127:
> /usr/src/kernels/3.13.0-0.rc7.31.sa2.k32v1.aarch64.debug/arch/arm64/include/
> asm/xen/hypervisor.h:1:50: fatal error:
> ../../arm/include/asm/xen/hypervisor.h: No such file or directory
>  #include <../../arm/include/asm/xen/hypervisor.h>
>                                                   ^
> compilation terminated.
> make[1]: ***
> [/tmp/stapSX3Lm0/stap_a9d73a0f48c94b332f8efe28f9c0739d_104773_src.o] Error 1
> make: *** [_module_/tmp/stapSX3Lm0] Error 2
> WARNING: kbuild exited with status: 2
> Pass 4: compilation failed.  [man error::pass4]
> ====
> 
> So that looks like a kernel problem, not a systemtap problem.


Right, This is rhbz1094366, known and fixed there in c#2

> 
> On your kernel, something odd is going on if things like __NR_open aren't
> defined. Can you try the following patch and see if that fixes things?
> 
> ====
> diff --git a/runtime/linux/compat_unistd.h b/runtime/linux/compat_unistd.h
> index 25ca1bd..11abc26 100644
> --- a/runtime/linux/compat_unistd.h
> +++ b/runtime/linux/compat_unistd.h
> @@ -11,6 +11,8 @@
>  #ifndef _COMPAT_UNISTD_H_
>  #define _COMPAT_UNISTD_H_
>  
> +#include <linux/unistd.h>
> +
>  #if defined(__x86_64__)
>  
>  // On older kernels (like RHEL5), we have to define our own 32-bit
> ====

applied to /usr/share/systemtap/runtime/linux/compat_unistd.h, but no change
compared to c#6.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #9 from David Smith <dsmith at redhat dot com> ---
OK, I"ve got the systemtap problems fixed in commit 293523b. However, that
kernel doesn't have kprobes:

====
# stap -vp4 -e 'probe syscall.open { printf("hi\n") }'
Pass 1: parsed user script and 104 library script(s) using
149268virt/29880res/2940shr/27308data kb, in 380usr/10sys/399real ms.
Pass 2: analyzed script: 2 probe(s), 1 function(s), 40 embed(s), 0 global(s)
using 180248virt/61760res/3812shr/58288data kb, in 1440usr/160sys/1600real ms.
Pass 3: translated to C into
"/tmp/stapewvvDK/stap_62dcdbc1e584020a6a19f7cc43b29fdc_24567_src.c" using
180248virt/62096res/4148shr/58288data kb, in 20usr/110sys/136real ms.
/tmp/stapewvvDK/stap_62dcdbc1e584020a6a19f7cc43b29fdc_24567_src.c:1567:2:
error: #error "Need CONFIG_KPROBES!"
 #error "Need CONFIG_KPROBES!"
  ^
make[1]: ***
[/tmp/stapewvvDK/stap_62dcdbc1e584020a6a19f7cc43b29fdc_24567_src.o] Error 1
make: *** [_module_/tmp/stapewvvDK] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_62dcdbc1e584020a6a19f7cc43b29fdc_24567.ko" in
9880usr/1290sys/12392real ms.
Pass 4: compilation failed.  [man error::pass4]
====

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[aram.h at mgk dot ro]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

Aram Hăvărneanu <aram.h at mgk dot ro> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aram.h at mgk dot ro

--- Comment #10 from Aram Hăvărneanu <aram.h at mgk dot ro> ---
I am having the same problem as the one describe in comment 7.

In file included from ./arch/arm64/include/asm/dma-mapping.h:27:0,
                 from include/linux/dma-mapping.h:82,
                 from include/linux/skbuff.h:34,
                 from include/linux/netlink.h:6,
                 from
/tmp/stap85zy5C/stap_6c1847e871322a4ef474645ee4b86ee9_606063_src.c:184:
./arch/arm64/include/asm/xen/hypervisor.h:1:50: fatal error:
../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
                                                  ^
compilation terminated.

I am running the upstream 3.19.0 kernel.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #11 from David Smith <dsmith at redhat dot com> ---
(In reply to Aram Hăvărneanu from comment #10)
> I am having the same problem as the one describe in comment 7.
> 
> In file included from ./arch/arm64/include/asm/dma-mapping.h:27:0,
>                  from include/linux/dma-mapping.h:82,
>                  from include/linux/skbuff.h:34,
>                  from include/linux/netlink.h:6,
>                  from
> /tmp/stap85zy5C/stap_6c1847e871322a4ef474645ee4b86ee9_606063_src.c:184:
> ./arch/arm64/include/asm/xen/hypervisor.h:1:50: fatal error:
> ../../arm/include/asm/xen/hypervisor.h: No such file or directory
>  #include <../../arm/include/asm/xen/hypervisor.h>
>                                                   ^
> compilation terminated.
> 
> I am running the upstream 3.19.0 kernel.

As mentioned in comment #8 this is a kernel problem, not a systemtap problem.
This is rhbz1094366 <https://bugzilla.redhat.com/show_bug.cgi?id=1094366>. It
was fixed at one point, perhaps it has gotten broken again.

It isn't systemtap's fault that when you include a kernel header, one of the
other kernel headers it includes is a file that doesn't exist.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[aram.h at mgk dot ro]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #12 from Aram Hăvărneanu <aram.h at mgk dot ro> ---
Yes, of course it is not SystemTap's fault that the kernel is broken. I can't
see https://bugzilla.redhat.com/show_bug.cgi?id=1094366 though (access denied),
so I can't patch my upstream kernel locally.

In short, it's not SystemTap's fault, but you can't run SystemTap on arm64
today (except, perhaps, on RedHat kernel; who knows, I can't read that bug). So
I posted so other people who hit this know what to expect.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[dsmith at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #13 from David Smith <dsmith at redhat dot com> ---
(In reply to Aram Hăvărneanu from comment #12)
> Yes, of course it is not SystemTap's fault that the kernel is broken. I
> can't see https://bugzilla.redhat.com/show_bug.cgi?id=1094366 though (access
> denied), so I can't patch my upstream kernel locally.

Sorry, I didn't realize that was an internal bug. I'm not sure why it is. I'll
see if it can be made public. Long story short, the bug adds the following text
to the spec file:

+%ifarch aarch64
+ cp -a --parents arch/arm/include/asm/xen
$RPM_BUILD_ROOT/lib/modules/$KernelVer/build/
+%endif

> In short, it's not SystemTap's fault, but you can't run SystemTap on arm64
> today (except, perhaps, on RedHat kernel; who knows, I can't read that bug).
> So I posted so other people who hit this know what to expect.

Hopefully that above command should do the trick for you (after making it fit
your environment).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug translator/16914] probe syscall.* fails with compilation error

[aram.h at mgk dot ro]

https://sourceware.org/bugzilla/show_bug.cgi?id=16914

--- Comment #14 from Aram Hăvărneanu <aram.h at mgk dot ro> ---
Thanks. For reference, this is the fix for debian-bases systems:
http://sprunge.us/GiQW

-- 
You are receiving this mail because:
You are the assignee for the bug.