[BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[agentzh]

Hello!

The recent git master HEAD of systemtap does not work at all on my Fedora 17:

    $ uname -a
    Linux w530 3.6.7-4.fc17.x86_64 #1 SMP Tue Nov 20 19:40:01 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux

I'm using the latest kernel from the official yum repository of Fedora 17.

The systemtap build is compiled from the latest git master HEAD
(commit 6b378b7c40b):

    $ gcc --version
    gcc (GCC) 4.7.2 20120921 (Red Hat 4.7.2-2)
    Copyright (C) 2012 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    Even hello world does not work because "stap" quits at startup:

    $  ./configure --prefix=/opt/systemtap --disable-docs
--disable-publican --disable-refdocs && make -j8
    ...

    $ sudo make install
    ...

    $ PATH=/opt/systemtap/bin:$PATH stap --version
    Systemtap translator/driver (version 2.1/0.154, commit
release-2.0-108-g6b378b7)
    Copyright (C) 2005-2012 Red Hat, Inc. and others
    This is free software; see the source for copying conditions.
    enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR
TR1_UNORDERED_MAP NLS

    $ PATH=/opt/systemtap/bin:$PATH stap -v -e 'probe begin {
println("hello") exit() }'

Yes, no output at all. Using strace to inspect it:

    $ PATH=/opt/systemtap/bin:$PATH strace stap -v -e 'probe begin {
println("hello") exit() }'
    ...
    brk(0x10e9000)                          = 0x10e9000
    read(3, "00000000\tacpi_unload_table_id\tvm"..., 8191) = 8191
    read(3, "EXPORT_SYMBOL\n0x00000000\tvb2_ioc"..., 8191) = 8191
    read(3, "0000\tunregister_sound_dsp\tsound/"..., 8191) = 1009
    read(3, "", 8191)                       = 0
    close(3)                                = 0
    open("/lib/modules/3.6.7-4.fc17.x86_64/build/System.map",
O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/boot/System.map-3.6.7-4.fc17.x86_64", O_RDONLY) = -1 EACCES
(Permission denied)
    rt_sigprocmask(SIG_BLOCK, [HUP INT PIPE TERM], [], 8) = 0
    clone(child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f0420eccad0) = 9106
    rt_sigprocmask(SIG_BLOCK, [HUP INT PIPE TERM], [HUP INT PIPE TERM], 8) = 0
    rt_sigprocmask(SIG_SETMASK, [HUP INT PIPE TERM], NULL, 8) = 0
    wait4(9106, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 9106
    --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=9106,
si_status=0, si_utime=0, si_stime=0} ---
    rt_sigprocmask(SIG_BLOCK, [HUP INT PIPE TERM], [HUP INT PIPE TERM], 8) = 0
    rt_sigprocmask(SIG_SETMASK, [HUP INT PIPE TERM], NULL, 8) = 0
    rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
    exit_group(1)                           = ?
    +++ exited with 1 +++

It seems that SIGCHLD happens there.

Using the systemtap 2.0 release tarball works for me:

    $ PATH=/opt/systemtap20/bin:$PATH stap --version
    Systemtap translator/driver (version 2.0/0.154, non-git sources)
    Copyright (C) 2005-2012 Red Hat, Inc. and others
    This is free software; see the source for copying conditions.
    enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR
TR1_UNORDERED_MAP NLS

    $ PATH=/opt/systemtap20/bin:$PATH stap -v -e 'probe begin {
println("hello") exit() }'
    Pass 1: parsed user script and 89 library script(s) using
202416virt/21940res/2852shr/19736data kb, in 90usr/20sys/450real ms.
    Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0
global(s) using 202944virt/22712res/2952shr/20264data kb, in
10usr/0sys/45real ms.
    Pass 3: translated to C into
"/tmp/stappbpQm5/stap_3de736424383f206ac40c9667b3049ae_963_src.c"
using 202944virt/23016res/3212shr/20264data kb, in 0usr/0sys/0real ms.
    Pass 4: compiled C into
"stap_3de736424383f206ac40c9667b3049ae_963.ko" in
1120usr/330sys/1596real ms.
    Pass 5: starting run.
    hello
    Pass 5: run completed in 10usr/20sys/348real ms.

If you need more information to fix this, please let me know.

I really hope that the git master HEAD can be a little more stable ;)

Best regards,
-agentzh

Re: [BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[agentzh]

Hello!

On Mon, Nov 26, 2012 at 2:54 PM, agentzh wrote:
>     open("/lib/modules/3.6.7-4.fc17.x86_64/build/System.map",
> O_RDONLY) = -1 ENOENT (No such file or directory)
>     open("/boot/System.map-3.6.7-4.fc17.x86_64", O_RDONLY) = -1 EACCES
> (Permission denied)

Okay...from the strace output given by myself, it seems that it was
just because stap failed to read the file
/boot/System.map-3.6.7-4.fc17.x86_64:

    $ ls -lh /boot/System.map-3.6.7-4.fc17.x86_64
    -rw-------. 1 root root 2.4M Nov 20 11:43
/boot/System.map-3.6.7-4.fc17.x86_64

Because I'm using my own non-root account (belonging to the stapdev
and stapusr groups), stap fails to read this file. Fixing the file
permissions make stap work for me again:

    $ PATH=/opt/systemtap/bin:$PATH stap -v -e 'probe begin {
println("hello") exit() }'
    Pass 1: parsed user script and 91 library script(s) using
205804virt/25364res/2860shr/23128data kb, in 90usr/0sys/94real ms.
    Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0
global(s) using 206332virt/26152res/2980shr/23656data kb, in
0usr/0sys/2real ms.
    Pass 3: using cached
/home/agentzh/.systemtap/cache/02/stap_0243d4699068e62d2cf9e66a07758f6b_950.c
    Pass 4: using cached
/home/agentzh/.systemtap/cache/02/stap_0243d4699068e62d2cf9e66a07758f6b_950.ko
    Pass 5: starting run.
    hello
    Pass 5: run completed in 0usr/20sys/322real ms.

Maybe stap should give an error message here instead of just crashing silently?

Best regards,
-agentzh

Re: [BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[Josh Stone]

On 11/26/2012 03:05 PM, agentzh wrote:
> Hello!
> 
> On Mon, Nov 26, 2012 at 2:54 PM, agentzh wrote:
>>     open("/lib/modules/3.6.7-4.fc17.x86_64/build/System.map",
>> O_RDONLY) = -1 ENOENT (No such file or directory)
>>     open("/boot/System.map-3.6.7-4.fc17.x86_64", O_RDONLY) = -1 EACCES
>> (Permission denied)
> 
> Okay...from the strace output given by myself, it seems that it was
> just because stap failed to read the file
> /boot/System.map-3.6.7-4.fc17.x86_64:

The /boot/System.map is used as a fallback, but I'm actually more
surprised at the ENOENT for the /lib/modules/ path.  That ought to be
present as part of the kernel-devel package.  In fact, the strace read()
lines you pasted before look like they're showing good data from
Module.symvers, also part of kernel-devel.

Any idea why your kernel-devel would seem incomplete?  I have the same
kernel-devel-3.6.7-4.fc17.x86_64 installed, and for me it contains both
Module.symvers and System.map.

> Maybe stap should give an error message here instead of just crashing silently?

There actually is an error message, guarded by verbose>3, which is way
too quiet.  This is moved code that used to live in a non-critical path.
 (and perhaps it shouldn't be treated as critical here either, but we
can consider that separately.)  So yes, we should always print a message
when something errors out, and I'll make this do so.

Josh

Re: [BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[agentzh]

Hello!

On Mon, Nov 26, 2012 at 3:32 PM, Josh Stone wrote:
> Any idea why your kernel-devel would seem incomplete?  I have the same
> kernel-devel-3.6.7-4.fc17.x86_64 installed, and for me it contains both
> Module.symvers and System.map.
>

I have no idea why my installation is incomplete. I just run "yum
update" regularly.

There is Module.symvers but no System.map under
/lib/modules/3.6.7-4.fc17.x86_64/build/.

Any suggestions to track this further?

> So yes, we should always print a message
> when something errors out, and I'll make this do so.
>

That's good :)

Thanks!
-agentzh

Re: [BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[Josh Stone]

On 11/27/2012 03:58 PM, agentzh wrote:
> Hello!
> 
> On Mon, Nov 26, 2012 at 3:32 PM, Josh Stone wrote:
>> Any idea why your kernel-devel would seem incomplete?  I have the same
>> kernel-devel-3.6.7-4.fc17.x86_64 installed, and for me it contains both
>> Module.symvers and System.map.
>>
> 
> I have no idea why my installation is incomplete. I just run "yum
> update" regularly.
> 
> There is Module.symvers but no System.map under
> /lib/modules/3.6.7-4.fc17.x86_64/build/.
> 
> Any suggestions to track this further?

Try to verify: rpm -vV kernel-devel-3.6.7-4.fc17.x86_64

I get a lot of timestamp (T) changes, but System.map is right there at
the top of the list without any verify flags set.

Do you have any third-party kernel modules?  Sometimes those play games
in the kernel build tree to insert themselves, and it could be that one
is playing this game badly.

If nothing else, "yum reinstall kernel-devel" should bring it back. :)


Josh

Re: [BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[agentzh]

Hello!

On Tue, Nov 27, 2012 at 5:52 PM, Josh Stone wrote:
>
> Try to verify: rpm -vV kernel-devel-3.6.7-4.fc17.x86_64
>
> I get a lot of timestamp (T) changes, but System.map is right there at
> the top of the list without any verify flags set.
>

Thank you for the tip!

I'm indeed getting the output line

    missing     /usr/src/kernels/3.6.7-4.fc17.x86_64/System.map

Weird.

> Do you have any third-party kernel modules?  Sometimes those play games
> in the kernel build tree to insert themselves, and it could be that one
> is playing this game badly.
>

No, I don't think I have such kernel modules :)

> If nothing else, "yum reinstall kernel-devel" should bring it back. :)
>

Unfortunately, this command does not work for me it seems:

    $ sudo yum reinstall kernel-devel
    Loaded plugins: auto-update-debuginfo, changelog, langpacks,
presto, refresh-packagekit
    updates-debuginfo/17/x86_64/metalink
                                     |  12 kB     00:00
    Installed package kernel-devel-3.6.6-1.fc17.x86_64 (from updates)
not available.
    Installed package kernel-devel-3.6.5-1.fc17.x86_64 (from updates)
not available.
    Installed package kernel-devel-3.6.7-4.fc17.x86_64 (from updates)
not available.
    Error: Nothing to do

I don't understand what it means :P

Thank you for your help!

Best regards,
-agentzh

Re: [BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[Frank Ch. Eigler]

agentzh <agentzh@gmail.com> writes:

> [...]
> Unfortunately, this command does not work for me it seems:
>
>     $ sudo yum reinstall kernel-devel
>     Loaded plugins: auto-update-debuginfo, changelog, langpacks,
> presto, refresh-packagekit
>     updates-debuginfo/17/x86_64/metalink
> [...]
>     Installed package kernel-devel-3.6.7-4.fc17.x86_64 (from updates)
> not available.
>     Error: Nothing to do

I think it means just that you're more than three versions behind the
current update of the kernel.  You could do a "yum -y update kernel\*",
or else find the exact kernel-devel version/arch/dist you need by 
searching through http://koji.fedoraproject.org/koji/packageinfo?packageID=8

- FChE

Re: [BUG] Systemtap git master HEAD fails on Fedora 17 x86_64 (kernel 3.6.7)

[agentzh]

Hello!

On Wed, Nov 28, 2012 at 5:32 AM, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> I think it means just that you're more than three versions behind the
> current update of the kernel.  You could do a "yum -y update kernel\*",
> or else find the exact kernel-devel version/arch/dist you need by
> searching through http://koji.fedoraproject.org/koji/packageinfo?packageID=8
>

Thank you for the help! It works for me now :)

Best regards,
-agentzh