Recent review of SystemTap test results on ARM running Fedora 15 hard float.

Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[William Cohen]

I reviewed the SystemTap testsuite failures for Fedora 15 hard float running on ARM.  You can see the recent ARM test results in dejazilla at:

http://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=40&_sort=1A&summary=&age=&rg=&tool=&variant=%3D%27armv7l-unknown-linux-gnu%27&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=


FAIL: backtrace-unwindsyms (0 0)
FAIL: bz6503 0 0
   bz6503.exp fails because no jffs2 or ext2 kernel modules.
	   Maybe use nfs and fat instead?

FAIL: cmd_parse15: eof
      cmd_parse.exp fail because avahi problems

FAIL: global_end (11)
      global_end.exp fails because "time = 0" is output by global_end2.stp

FAIL: gtod (0)
      gettimeofday_*() function do not work properly on arm

FAIL: ipaddr_IPv4_recvmsg find 'nc'
      doesn't seem to find /usr/bin/nc, but nc rpm has been installed on machine

FAIL: OVERLOAD2 no expected error
      Looks like this is caused by arm always returning 0 for get_cycles().
      This failure is troubling because OVERLOAD stuff will not work.

FAIL: prcwildcard function
      failed because no user space probing is available

FAIL: systemtap.base/pt_user_mode.stp startup (eof)
FAIL: systemtap.examples/profiling/thread-times run
      failed because on startup unable to register probe:
      	      perf.type(1).config(0).sample(1000000)

FAIL: vma_vdsodefault
      vma_vdso.stp uses uaddr() which didn't work on arm.

FAIL: warnings (timeout)
FAIL: warnings (0)
      took too long to compile systemtap.base/warning.stp, about 2:30
      fixed in commit 9e6bcc7a2ee69f4211135900e99bad9e1abfc665

FAIL: int64 function arguments -- numeric
FAIL: int64 function arguments -- numeric --kelf --ignore-dwarf
      PR13466

FAIL: systemtap.examples/io/mbrwatch run
      ran test as normal user dd used to test this fails as a result

FAIL: systemtap.examples/memory/hw_watch_addr run
FAIL: systemtap.examples/memory/hw_watch_sym run
      not enough hardware breakpoints available:
      WARNING: Too many hardware breakpoint probes requested for arm (1 vs. 0)
`

FAIL: uprobeslibgcc-O3default-prelink-debug prelink ./libuprobeslibgcc-O3default-prelink-debug.so
FAIL: uprobeslibgcc-O3default-prelink-sep-debug prelink ./libuprobeslibgcc-O3default-prelink-sep-debug.so
FAIL: uprobeslibgcc-O3default-prelink-sep-debug-after prelink ./libuprobeslibgcc-O3default-prelink-sep-debug-after.so
      uprobes not available on arm

FAIL: buildok/eighteen.stp
      unable to find kernel.function("*audit_getname@kernel/auditsc.c")
      CONFIG_AUDITSYSCALL is not enabled on arm fedora 15 kernels,
      something is diabling.


FAIL: buildok/process_test.stp
      unable to find $signr for:
        tapset/signal.stp:606:15
			:607:33


FAIL: buildok/syscalls-arch-detailed.stp
      unable to probe syscall.sigaltstack.return
      ARM doesn't have a sys_sigaltstack() function in the kernel.

FAIL: buildok/tcp-all-probes.stp
FAIL: buildok/tcp-detailed.stp
      unable to get identifier '$optname' tapset/tcp.stp:369:27

FAIL: buildok/twentyeightprime.stp
      CONFIG_UTRACE not available (seems like that should be UNTESTED)

FAIL: semok/thirtynine.stp
      can't access identifier '$prev' at .../semok/thirtynine.stp:6:40
FAIL: semok/thirtysix.stp
WARNING: Can't parse SDT_V3 operand 'r0': identifier '$arg1' at /media/greatpla\
ins/wcohen/systemtap_write/systemtap/testsuite/semok/thirtysix.stp:11:53

FAIL: systemtap.stress/current.stp compilation
      can't probe kernel.function("__switch_to").call
      need to adapt for arm

FAIL: 32-bit * nd_syscall
     ARM registers.stp can only read the first 4 arguments in registers.
     fifth/sixth arguments are problems on ARM, so these tests fail.

FAIL: unprivileged embedded C: no embedded C: --unprivileged: inet_get_ip_source(long)
FAIL: unprivileged embedded C: no embedded C: --unprivileged: get_ip(long)

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[David Smith]

On 02/23/2012 01:06 PM, William Cohen wrote:

> I reviewed the SystemTap testsuite failures for Fedora 15 hard float running on ARM.  You can see the recent ARM test results in dejazilla at:
> 
> http://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=40&_sort=1A&summary=&age=&rg=&tool=&variant=%3D%27armv7l-unknown-linux-gnu%27&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=


Here are some notes on certain tests:

> FAIL: ipaddr_IPv4_recvmsg find 'nc'
>       doesn't seem to find /usr/bin/nc, but nc rpm has been installed on machine


That's odd.  That test tries to find nc by running "which nc".  If that
succeeds, it assumes nc is there.  Does "which nc" find /usr/bin/nc and
return a 0 exit code?

> FAIL: vma_vdsodefault
>       vma_vdso.stp uses uaddr() which didn't work on arm.


Hmm, can you tell in systemtap.log what part of 'uaddr()' didn't work?

>

> FAIL: uprobeslibgcc-O3default-prelink-debug prelink ./libuprobeslibgcc-O3default-prelink-debug.so
> FAIL: uprobeslibgcc-O3default-prelink-sep-debug prelink ./libuprobeslibgcc-O3default-prelink-sep-debug.so
> FAIL: uprobeslibgcc-O3default-prelink-sep-debug-after prelink ./libuprobeslibgcc-O3default-prelink-sep-debug-after.so
>       uprobes not available on arm


This is odd.  I would have thought the testsuite would have realized
this platform didn't have uprobes, and not run the test at all.  Which
.exp is this, exelib.exp?

> FAIL: buildok/syscalls-arch-detailed.stp
>       unable to probe syscall.sigaltstack.return
>       ARM doesn't have a sys_sigaltstack() function in the kernel.


That's PR13480.

> FAIL: buildok/tcp-all-probes.stp
> FAIL: buildok/tcp-detailed.stp
>       unable to get identifier '$optname' tapset/tcp.stp:369:27


Hmm, this one looks like a debuginfo problem.  Could you show me the
output of:

# stap -L 'kernel.function("tcp_setsockopt")

> FAIL: buildok/twentyeightprime.stp
>       CONFIG_UTRACE not available (seems like that should be UNTESTED)


Actually it should be KFAIL.  I'll fix this.

> 
> FAIL: semok/thirtynine.stp
>       can't access identifier '$prev' at .../semok/thirtynine.stp:6:40


I'd guess this is a debuginfo problem.

> FAIL: systemtap.stress/current.stp compilation
>       can't probe kernel.function("__switch_to").call
>       need to adapt for arm


File a PR on this one.

> FAIL: unprivileged embedded C: no embedded C: --unprivileged: inet_get_ip_source(long)
> FAIL: unprivileged embedded C: no embedded C: --unprivileged: get_ip(long)


I've always been fuzzy on what exactly these errors mean.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[William Cohen]

On 02/23/2012 03:54 PM, David Smith wrote:
> On 02/23/2012 01:06 PM, William Cohen wrote:

Hi David,

Thanks for the comments/questions.

> 
>> I reviewed the SystemTap testsuite failures for Fedora 15 hard float running on ARM.  You can see the recent ARM test results in dejazilla at:
>>
>> http://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=40&_sort=1A&summary=&age=&rg=&tool=&variant=%3D%27armv7l-unknown-linux-gnu%27&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=
> 
> 
> Here are some notes on certain tests:
> 
>> FAIL: ipaddr_IPv4_recvmsg find 'nc'
>>       doesn't seem to find /usr/bin/nc, but nc rpm has been installed on machine

I need to pay attention to editing. I installed nc on the machine after I realized why the test was failing, but didn't write things correctly after trying stuff.  The test passes after installing the nc rpm.

> 
> 
> That's odd.  That test tries to find nc by running "which nc".  If that
> succeeds, it assumes nc is there.  Does "which nc" find /usr/bin/nc and
> return a 0 exit code?
> 
>> FAIL: vma_vdsodefault
>>       vma_vdso.stp uses uaddr() which didn't work on arm.
> 
> 
> Hmm, can you tell in systemtap.log what part of 'uaddr()' didn't work?

The log wasn't too helpful. When running the test by hand.  Looks like umodename(uaddrr()) == "<unknown>", so getting from

	  printf("%s@%x unknown\n", name, uaddr());

>>
> 
>> FAIL: uprobeslibgcc-O3default-prelink-debug prelink ./libuprobeslibgcc-O3default-prelink-debug.so
>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug prelink ./libuprobeslibgcc-O3default-prelink-sep-debug.so
>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug-after prelink ./libuprobeslibgcc-O3default-prelink-sep-debug-after.so
>>       uprobes not available on arm
> 
> 
> This is odd.  I would have thought the testsuite would have realized
> this platform didn't have uprobes, and not run the test at all.  Which
> .exp is this, exelib.exp?

Yes, it appears to be running:

systemtap.exelib/exelib.exp ...


> 
>> FAIL: buildok/syscalls-arch-detailed.stp
>>       unable to probe syscall.sigaltstack.return
>>       ARM doesn't have a sys_sigaltstack() function in the kernel.
> 
> 
> That's PR13480.
> 
>> FAIL: buildok/tcp-all-probes.stp
>> FAIL: buildok/tcp-detailed.stp
>>       unable to get identifier '$optname' tapset/tcp.stp:369:27
> 
> 
> Hmm, this one looks like a debuginfo problem.  Could you show me the
> output of:
> 
> # stap -L 'kernel.function("tcp_setsockopt")

$ ../install/bin/stap -L 'kernel.function("tcp_setsockopt")'
kernel.function("tcp_setsockopt@net/ipv4/tcp.c:2406") $optlen:unsigned int $sk:struct sock*

> 
>> FAIL: buildok/twentyeightprime.stp
>>       CONFIG_UTRACE not available (seems like that should be UNTESTED)
> 
> 
> Actually it should be KFAIL.  I'll fix this.
> 
>>
>> FAIL: semok/thirtynine.stp
>>       can't access identifier '$prev' at .../semok/thirtynine.stp:6:40
> 
> 
> I'd guess this is a debuginfo problem.
> 
>> FAIL: systemtap.stress/current.stp compilation
>>       can't probe kernel.function("__switch_to").call
>>       need to adapt for arm
> 
> 
> File a PR on this one.


Looks very similar to PR4331 - systemtap.stress/current.stp need to be updated for s390x
Made clone: PR13734 - systemtap.stress/current.stp need to be updated for arm

> 
>> FAIL: unprivileged embedded C: no embedded C: --unprivileged: inet_get_ip_source(long)
>> FAIL: unprivileged embedded C: no embedded C: --unprivileged: get_ip(long)
> 
> 
> I've always been fuzzy on what exactly these errors mean.

I wasn't too clear on the last two, so I didn't comment on them either. :/

-Will

 

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[David Smith]

On 02/23/2012 02:54 PM, David Smith wrote:

> On 02/23/2012 01:06 PM, William Cohen wrote:
> 
>> FAIL: buildok/twentyeightprime.stp
>>       CONFIG_UTRACE not available (seems like that should be UNTESTED)
> 
> Actually it should be KFAIL.  I'll fix this.


Fixed in commit b0a5a91.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[David Smith]

On 02/23/2012 04:06 PM, William Cohen wrote:

> On 02/23/2012 03:54 PM, David Smith wrote:
>> On 02/23/2012 01:06 PM, William Cohen wrote:
> 
> Hi David,
> 
> Thanks for the comments/questions.
> 
>>
>>> I reviewed the SystemTap testsuite failures for Fedora 15 hard float running on ARM.  You can see the recent ARM test results in dejazilla at:
>>>
>>> http://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=40&_sort=1A&summary=&age=&rg=&tool=&variant=%3D%27armv7l-unknown-linux-gnu%27&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=
>>
>>
>> Here are some notes on certain tests:
>>
>>> FAIL: ipaddr_IPv4_recvmsg find 'nc'
>>>       doesn't seem to find /usr/bin/nc, but nc rpm has been installed on machine
> 
> I need to pay attention to editing. I installed nc on the machine after I realized why the test was failing, but didn't write things correctly after trying stuff.  The test passes after installing the nc rpm.


Great, I thought that test should have worked there.

>>> FAIL: vma_vdsodefault
>>>       vma_vdso.stp uses uaddr() which didn't work on arm.
>>
>>
>> Hmm, can you tell in systemtap.log what part of 'uaddr()' didn't work?
> 
> The log wasn't too helpful. When running the test by hand.  Looks like umodename(uaddrr()) == "<unknown>", so getting from
> 
> 	  printf("%s@%x unknown\n", name, uaddr());


Hmm, I'll try looking at this one some more.

>>> FAIL: uprobeslibgcc-O3default-prelink-debug prelink ./libuprobeslibgcc-O3default-prelink-debug.so
>>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug prelink ./libuprobeslibgcc-O3default-prelink-sep-debug.so
>>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug-after prelink ./libuprobeslibgcc-O3default-prelink-sep-debug-after.so
>>>       uprobes not available on arm
>>
>>
>> This is odd.  I would have thought the testsuite would have realized
>> this platform didn't have uprobes, and not run the test at all.  Which
>> .exp is this, exelib.exp?
> 
> Yes, it appears to be running:
> 
> systemtap.exelib/exelib.exp ...

Ah, I see what this one is doing now.  I'll bet prelink isn't available
on arm.  This testcase builds an exe, prelinks it, then does skip the
actual systemtap tests if !uprobes.

Is prelink present on your arm system?

Since prelink isn't present on ia64 either, it might be worthwhile to
have this testcase check for prelink.

>>> FAIL: buildok/tcp-all-probes.stp
>>> FAIL: buildok/tcp-detailed.stp
>>>       unable to get identifier '$optname' tapset/tcp.stp:369:27
>>
>>
>> Hmm, this one looks like a debuginfo problem.  Could you show me the
>> output of:
>>
>> # stap -L 'kernel.function("tcp_setsockopt")
> 
> $ ../install/bin/stap -L 'kernel.function("tcp_setsockopt")'
> kernel.function("tcp_setsockopt@net/ipv4/tcp.c:2406") $optlen:unsigned int $sk:struct sock*

Yep, looks like bad debuginfo.  That's missing several options, here's
the definition:

static int do_tcp_setsockopt(struct sock *sk, int level,
		int optname, char __user *optval, unsigned int optlen)

>>> FAIL: systemtap.stress/current.stp compilation

>>>       can't probe kernel.function("__switch_to").call
>>>       need to adapt for arm
>>
>>
>> File a PR on this one.
> 
> 
> Looks very similar to PR4331 - systemtap.stress/current.stp need to be updated for s390x
> Made clone: PR13734 - systemtap.stress/current.stp need to be updated for arm

Great.

>>> FAIL: unprivileged embedded C: no embedded C: --unprivileged: inet_get_ip_source(long)
>>> FAIL: unprivileged embedded C: no embedded C: --unprivileged: get_ip(long)
>>
>>
>> I've always been fuzzy on what exactly these errors mean.
> 
> I wasn't too clear on the last two, so I didn't comment on them either. :/


-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[David Smith]

On 02/23/2012 04:55 PM, David Smith wrote:

> On 02/23/2012 04:06 PM, William Cohen wrote:
> 
>> On 02/23/2012 03:54 PM, David Smith wrote:
>>> On 02/23/2012 01:06 PM, William Cohen wrote:
>>
>> Hi David,
>>
>> Thanks for the comments/questions.
>>
>>>
>>>> I reviewed the SystemTap testsuite failures for Fedora 15 hard float running on ARM.  You can see the recent ARM test results in dejazilla at:
>>>>
>>>> http://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=40&_sort=1A&summary=&age=&rg=&tool=&variant=%3D%27armv7l-unknown-linux-gnu%27&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=
>>>
>>>


>>>> FAIL: vma_vdsodefault
>>>>       vma_vdso.stp uses uaddr() which didn't work on arm.
>>>
>>>
>>> Hmm, can you tell in systemtap.log what part of 'uaddr()' didn't work?
>>
>> The log wasn't too helpful. When running the test by hand.  Looks like umodename(uaddrr()) == "<unknown>", so getting from
>>
>> 	  printf("%s@%x unknown\n", name, uaddr());
> 
> 
> Hmm, I'll try looking at this one some more.


The first thing we need to do here is figure out which is really
failing, uaddr() or umodname().  Could you change the test to just print
out the return value of uaddr() to see what it is returning?

>>>> FAIL: uprobeslibgcc-O3default-prelink-debug prelink ./libuprobeslibgcc-O3default-prelink-debug.so
>>>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug prelink ./libuprobeslibgcc-O3default-prelink-sep-debug.so
>>>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug-after prelink ./libuprobeslibgcc-O3default-prelink-sep-debug-after.so
>>>>       uprobes not available on arm
>>>
>>>
>>> This is odd.  I would have thought the testsuite would have realized
>>> this platform didn't have uprobes, and not run the test at all.  Which
>>> .exp is this, exelib.exp?
>>
>> Yes, it appears to be running:
>>
>> systemtap.exelib/exelib.exp ...
> 
> Ah, I see what this one is doing now.  I'll bet prelink isn't available
> on arm.  This testcase builds an exe, prelinks it, then does skip the
> actual systemtap tests if !uprobes.
> 
> Is prelink present on your arm system?
> 
> Since prelink isn't present on ia64 either, it might be worthwhile to
> have this testcase check for prelink.


Commit 164901b changes exelib.exp to only run the prelink tests if
prelink exists on the system.  Try the test again and see if it works
better.

-- 

David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[William Cohen]

On 02/24/2012 11:03 AM, David Smith wrote:
> On 02/23/2012 04:55 PM, David Smith wrote:
> 
>> On 02/23/2012 04:06 PM, William Cohen wrote:
>>
>>> On 02/23/2012 03:54 PM, David Smith wrote:
>>>> On 02/23/2012 01:06 PM, William Cohen wrote:
>>>
>>> Hi David,
>>>
>>> Thanks for the comments/questions.
>>>
>>>>
>>>>> I reviewed the SystemTap testsuite failures for Fedora 15 hard float running on ARM.  You can see the recent ARM test results in dejazilla at:
>>>>>
>>>>> http://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=40&_sort=1A&summary=&age=&rg=&tool=&variant=%3D%27armv7l-unknown-linux-gnu%27&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=
>>>>
>>>>
> 
> 
>>>>> FAIL: vma_vdsodefault
>>>>>       vma_vdso.stp uses uaddr() which didn't work on arm.
>>>>
>>>>
>>>> Hmm, can you tell in systemtap.log what part of 'uaddr()' didn't work?
>>>
>>> The log wasn't too helpful. When running the test by hand.  Looks like umodename(uaddrr()) == "<unknown>", so getting from
>>>
>>> 	  printf("%s@%x unknown\n", name, uaddr());
>>
>>
>> Hmm, I'll try looking at this one some more.
> 
> 
> The first thing we need to do here is figure out which is really
> failing, uaddr() or umodname().  Could you change the test to just print
> out the return value of uaddr() to see what it is returning?

Added the following to the vma_vdso.stp just inside the (target() == pid())

+      printf("%s umodname(uaddr()) = umodname(%x) = %s\n",
+         name, uaddr(), umodname(uaddr()));


$ ../install/bin/stap  ../systemtap/testsuite/systemtap.base/vma_vdso.stp -c testsuite/vma_vdsodefault.exe 
clock_gettime umodname(uaddr()) = umodname(401096ec) = <unknown>
clock_gettime@401096ec unknown
getuid umodname(uaddr()) = umodname(40254e10) = <unknown>
getuid@40254e10 unknown
getuid umodname(uaddr()) = umodname(40221f5c) = <unknown>
getuid@40221f5c unknown

Below is the maps for the process to see where those addresses would map.  The results of uaddr() do not look correct.


$ more /proc/8622/maps
00008000-00009000 r-xp 00000000 08:10 3540441    /media/greatplains/wcohen/syste
mtap_write/obj/testsuite/vma_vdsodefault.exe
00010000-00011000 rw-p 00000000 08:10 3540441    /media/greatplains/wcohen/syste
mtap_write/obj/testsuite/vma_vdsodefault.exe
40000000-4001d000 r-xp 00000000 08:01 1815738    /lib/ld-2.14.1.so
4001d000-40020000 rw-p 00000000 00:00 0 
40024000-40025000 r--p 0001c000 08:01 1815738    /lib/ld-2.14.1.so
40025000-40026000 rw-p 0001d000 08:01 1815738    /lib/ld-2.14.1.so
40032000-40038000 r-xp 00000000 08:01 1815726    /lib/librt-2.14.1.so
40038000-4003f000 ---p 00006000 08:01 1815726    /lib/librt-2.14.1.so
4003f000-40040000 r--p 00005000 08:01 1815726    /lib/librt-2.14.1.so
40040000-40041000 rw-p 00006000 08:01 1815726    /lib/librt-2.14.1.so
40041000-400ac000 r-xp 00000000 08:01 1815692    /lib/libm-2.14.1.so
400ac000-400b3000 ---p 0006b000 08:01 1815692    /lib/libm-2.14.1.so
400b3000-400b4000 r--p 0006a000 08:01 1815692    /lib/libm-2.14.1.so
400b4000-400b5000 rw-p 0006b000 08:01 1815692    /lib/libm-2.14.1.so
400b5000-401e3000 r-xp 00000000 08:01 1815601    /lib/libc-2.14.1.so
401e3000-401ea000 ---p 0012e000 08:01 1815601    /lib/libc-2.14.1.so
401ea000-401ec000 r--p 0012d000 08:01 1815601    /lib/libc-2.14.1.so
401ec000-401ed000 rw-p 0012f000 08:01 1815601    /lib/libc-2.14.1.so
401ed000-401f0000 rw-p 00000000 00:00 0 
401f0000-401fa000 r-xp 00000000 08:01 1815713    /lib/libgcc_s-4.6.1-20110908.so
.1
401fa000-40201000 ---p 0000a000 08:01 1815713    /lib/libgcc_s-4.6.1-20110908.so
.1
40201000-40202000 rw-p 00009000 08:01 1815713    /lib/libgcc_s-4.6.1-20110908.so
.1
40202000-40216000 r-xp 00000000 08:01 1815734    /lib/libpthread-2.14.1.so
40216000-4021d000 ---p 00014000 08:01 1815734    /lib/libpthread-2.14.1.so
4021d000-4021e000 r--p 00013000 08:01 1815734    /lib/libpthread-2.14.1.so
4021e000-4021f000 rw-p 00014000 08:01 1815734    /lib/libpthread-2.14.1.so
4021f000-40221000 rw-p 00000000 00:00 0 
befdf000-bf000000 rw-p 00000000 00:00 0          [stack]
ffff0000-ffff1000 r-xp 00000000 00:00 0          [vectors]



> 
>>>>> FAIL: uprobeslibgcc-O3default-prelink-debug prelink ./libuprobeslibgcc-O3default-prelink-debug.so
>>>>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug prelink ./libuprobeslibgcc-O3default-prelink-sep-debug.so
>>>>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug-after prelink ./libuprobeslibgcc-O3default-prelink-sep-debug-after.so
>>>>>       uprobes not available on arm

...

> 
> Commit 164901b changes exelib.exp to only run the prelink tests if
> prelink exists on the system.  Try the test again and see if it works
> better.
> 

Prelink isn't currently on the ARM fedora. It might be there in the future.  Commit 164901b eliminates those failures on the ARM.

-Will

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[David Smith]

On 02/24/2012 12:26 PM, William Cohen wrote:

> On 02/24/2012 11:03 AM, David Smith wrote:
>> On 02/23/2012 04:55 PM, David Smith wrote:
>>
>>> On 02/23/2012 04:06 PM, William Cohen wrote:
>>>
>>>> On 02/23/2012 03:54 PM, David Smith wrote:
>>>>> On 02/23/2012 01:06 PM, William Cohen wrote:
>>>>
>>>> Hi David,
>>>>
>>>> Thanks for the comments/questions.
>>>>
>>>>>
>>>>>> I reviewed the SystemTap testsuite failures for Fedora 15 hard float running on ARM.  You can see the recent ARM test results in dejazilla at:
>>>>>>
>>>>>> http://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=40&_sort=1A&summary=&age=&rg=&tool=&variant=%3D%27armv7l-unknown-linux-gnu%27&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=
>>>>>
>>>>>
>>
>>
>>>>>> FAIL: vma_vdsodefault
>>>>>>       vma_vdso.stp uses uaddr() which didn't work on arm.
>>>>>
>>>>>
>>>>> Hmm, can you tell in systemtap.log what part of 'uaddr()' didn't work?
>>>>
>>>> The log wasn't too helpful. When running the test by hand.  Looks like umodename(uaddrr()) == "<unknown>", so getting from
>>>>
>>>> 	  printf("%s@%x unknown\n", name, uaddr());
>>>
>>>
>>> Hmm, I'll try looking at this one some more.
>>
>>
>> The first thing we need to do here is figure out which is really
>> failing, uaddr() or umodname().  Could you change the test to just print
>> out the return value of uaddr() to see what it is returning?
> 
> Added the following to the vma_vdso.stp just inside the (target() == pid())
> 
> +      printf("%s umodname(uaddr()) = umodname(%x) = %s\n",
> +         name, uaddr(), umodname(uaddr()));
> 
> 
> $ ../install/bin/stap  ../systemtap/testsuite/systemtap.base/vma_vdso.stp -c testsuite/vma_vdsodefault.exe 
> clock_gettime umodname(uaddr()) = umodname(401096ec) = <unknown>
> clock_gettime@401096ec unknown
> getuid umodname(uaddr()) = umodname(40254e10) = <unknown>
> getuid@40254e10 unknown
> getuid umodname(uaddr()) = umodname(40221f5c) = <unknown>
> getuid@40221f5c unknown
> 
> Below is the maps for the process to see where those addresses would map.  The results of uaddr() do not look correct.


...

Interesting.  So we got an address, but not a good address.  Here's the
source of uaddr():

function uaddr:long()
%{ /* pure */ /* myproc-unprivileged */
  struct pt_regs *uregs;


  if (CONTEXT->probe_flags & _STP_PROBE_STATE_USER_MODE)
    uregs = CONTEXT->uregs;
  else
    uregs = _stp_current_pt_regs();

  if (uregs)
    THIS->__retvalue = (int64_t) REG_IP(uregs);
  else
    THIS->__retvalue = 0;
%}

There are basically 3 possibilities of error here:

1) We're getting a bad uregs pointer from the context structure
2) We're getting a bad uregs pointer from the _stp_current_pt_regs().
3) REG_IP() isn't interpreting the good uregs pointer correctly

If you want to track this down further, first try to figure out where
we're getting uregs from.


>>>>>> FAIL: uprobeslibgcc-O3default-prelink-debug prelink ./libuprobeslibgcc-O3default-prelink-debug.so
>>>>>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug prelink ./libuprobeslibgcc-O3default-prelink-sep-debug.so
>>>>>> FAIL: uprobeslibgcc-O3default-prelink-sep-debug-after prelink ./libuprobeslibgcc-O3default-prelink-sep-debug-after.so
>>>>>>       uprobes not available on arm
> 
> ...
> 
>>
>> Commit 164901b changes exelib.exp to only run the prelink tests if
>> prelink exists on the system.  Try the test again and see if it works
>> better.
>>
> 
> Prelink isn't currently on the ARM fedora. It might be there in the future.  Commit 164901b eliminates those failures on the ARM.


prelink being available on ARM in the future shouldn't be a problem.
That commit does a file existence check on prelink, not an arch check.
So if prelink shows up it will get used.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[William Cohen]

On 02/24/2012 02:27 PM, David Smith wrote:
> On 02/24/2012 12:26 PM, William Cohen wrote:
> 
>> On 02/24/2012 11:03 AM, David Smith wrote:
>>> On 02/23/2012 04:55 PM, David Smith wrote:
>>>
>>>> On 02/23/2012 04:06 PM, William Cohen wrote:
>>>>
>>>>> On 02/23/2012 03:54 PM, David Smith wrote:
>>>>>> On 02/23/2012 01:06 PM, William Cohen wrote:
>>>>>
>>>>> Hi David,
>>>>>
>>>>> Thanks for the comments/questions.
>>>>>
>>>>>>
>>>>>>> I reviewed the SystemTap testsuite failures for Fedora 15 hard float running on ARM.  You can see the recent ARM test results in dejazilla at:
>>>>>>>
>>>>>>> http://web.elastic.org/~dejazilla/viewsummary.php?_offset=0&_limit=40&_sort=1A&summary=&age=&rg=&tool=&variant=%3D%27armv7l-unknown-linux-gnu%27&versions=&pass=&fail=&kpass=&kfail=&xpass=&xfail=&untested=&unresolved=&unsupported=&warning=&error=
>>>>>>
>>>>>>
>>>
>>>
>>>>>>> FAIL: vma_vdsodefault
>>>>>>>       vma_vdso.stp uses uaddr() which didn't work on arm.
>>>>>>
>>>>>>
>>>>>> Hmm, can you tell in systemtap.log what part of 'uaddr()' didn't work?
>>>>>
>>>>> The log wasn't too helpful. When running the test by hand.  Looks like umodename(uaddrr()) == "<unknown>", so getting from
>>>>>
>>>>> 	  printf("%s@%x unknown\n", name, uaddr());
>>>>
>>>>
>>>> Hmm, I'll try looking at this one some more.
>>>
>>>
>>> The first thing we need to do here is figure out which is really
>>> failing, uaddr() or umodname().  Could you change the test to just print
>>> out the return value of uaddr() to see what it is returning?
>>
>> Added the following to the vma_vdso.stp just inside the (target() == pid())
>>
>> +      printf("%s umodname(uaddr()) = umodname(%x) = %s\n",
>> +         name, uaddr(), umodname(uaddr()));
>>
>>
>> $ ../install/bin/stap  ../systemtap/testsuite/systemtap.base/vma_vdso.stp -c testsuite/vma_vdsodefault.exe 
>> clock_gettime umodname(uaddr()) = umodname(401096ec) = <unknown>
>> clock_gettime@401096ec unknown
>> getuid umodname(uaddr()) = umodname(40254e10) = <unknown>
>> getuid@40254e10 unknown
>> getuid umodname(uaddr()) = umodname(40221f5c) = <unknown>
>> getuid@40221f5c unknown
>>
>> Below is the maps for the process to see where those addresses would map.  The results of uaddr() do not look correct.
> 
> 
> ...
> 
> Interesting.  So we got an address, but not a good address.  Here's the
> source of uaddr():
> 
> function uaddr:long()
> %{ /* pure */ /* myproc-unprivileged */
>   struct pt_regs *uregs;
> 
> 
>   if (CONTEXT->probe_flags & _STP_PROBE_STATE_USER_MODE)
>     uregs = CONTEXT->uregs;
>   else
>     uregs = _stp_current_pt_regs();
> 
>   if (uregs)
>     THIS->__retvalue = (int64_t) REG_IP(uregs);
>   else
>     THIS->__retvalue = 0;
> %}
> 
> There are basically 3 possibilities of error here:
> 
> 1) We're getting a bad uregs pointer from the context structure
> 2) We're getting a bad uregs pointer from the _stp_current_pt_regs().
> 3) REG_IP() isn't interpreting the good uregs pointer correctly
> 
> If you want to track this down further, first try to figure out where
> we're getting uregs from.



It should be doing this code in kernel-space. Printed valude of user_mode() to verify.  Ruled out #1. It is getting information from _stp_current_pt_regs().

I started taking a look at the uaddr() and umodname() code to see where they are getting things.
See where REG_IP() coming from for arm, runtime/regs.h:

#elif defined (__arm__)

#define REG_IP(regs) regs->ARM_pc
#define REG_SP(regs) regs->ARM_sp
#define REG_LINK(regs) regs->ARM_lr

See ARM_pc defined in:

http://lxr.linux.no/#linux+v3.2.7/arch/arm/include/asm/ptrace.h#L114

I will do some more looking around why this doesn't work this evening.

-Will

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[Mark Wielaard]

On Fri, Feb 24, 2012 at 01:27:43PM -0600, David Smith wrote:
> On 02/24/2012 12:26 PM, William Cohen wrote:
> >>>>>> FAIL: vma_vdsodefault
> >>>>>>       vma_vdso.stp uses uaddr() which didn't work on arm.
> >>>>>
> >>>>> Hmm, can you tell in systemtap.log what part of 'uaddr()' didn't work?
> >>>>
> >>>> The log wasn't too helpful. When running the test by hand.  Looks like umodename(uaddrr()) == "<unknown>", so getting from
> >>>>
> >>>> 	  printf("%s@%x unknown\n", name, uaddr());
> >>>
> >>> Hmm, I'll try looking at this one some more.
> >>
> >> The first thing we need to do here is figure out which is really
> >> failing, uaddr() or umodname().  Could you change the test to just print
> >> out the return value of uaddr() to see what it is returning?
> > 
> > Added the following to the vma_vdso.stp just inside the (target() == pid())
> > 
> > +      printf("%s umodname(uaddr()) = umodname(%x) = %s\n",
> > +         name, uaddr(), umodname(uaddr()));
> > 
> > 
> > $ ../install/bin/stap  ../systemtap/testsuite/systemtap.base/vma_vdso.stp -c testsuite/vma_vdsodefault.exe 
> > clock_gettime umodname(uaddr()) = umodname(401096ec) = <unknown>
> > clock_gettime@401096ec unknown
> > getuid umodname(uaddr()) = umodname(40254e10) = <unknown>
> > getuid@40254e10 unknown
> > getuid umodname(uaddr()) = umodname(40221f5c) = <unknown>
> > getuid@40221f5c unknown
> > 
> > Below is the maps for the process to see where those addresses would map.  The results of uaddr() do not look correct.
> 
> 
> ...
> 
> Interesting.  So we got an address, but not a good address.

Note that this test explicitly tries to test an address has an known
module name, even if it is in the vdso, which is normally randomly
allocated in the process address space.

In the vma_vdso.c test program we see it uses a trick to make sure
we end up in the vdso (this depends on the implementation of clock_gettime
on the platform, so make sure on ARM this also results into actually
going into the vdso).

  /* Give an invalid clockid_t on purpose so the vdso has to call
     through to the kernel syscall. */

The testcase also tests the testuid call in two different ways.
Maye that is an easier case on ARM?

Then in vma_vdso.stp we see that the probes are in kernel space:
probe syscall.clock_gettime, syscall.getuid

It would surprise me if the uaddr() was really wrong, but it could
be. I suspect that the uaddr() is in the vdso and the vdso tracker
is broken on ARM.

A simpler test would be running this:

stap -e 'probe syscall.clock_gettime, syscall.getuidr
         { if (pid() == target()) printf("%s 0x%x in %s\n", name, uaddr(),
                                         umodname(uaddr())) }' \
     -c testsuite/vma_vdso-m64.exe

Which on x86_64 gives:

clock_gettime 0x7fff689ff9cf in vdso
getuid 0x3c5c0ed819 in /usr/lib64/libc-2.15.so
getuid 0x3c5c0bb007 in /usr/lib64/libc-2.15.so

Does that fail for all invocations on ARM?

If it is the vdso tracker then you want to take a look at
runtime/vma.c _stp_vma_match_vdso will give some debug output
with stap -DDEBUG_TASK_FINDER_VMA

Cheers,

Mark

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[William Cohen]

On 02/25/2012 02:39 PM, Mark Wielaard wrote:

> A simpler test would be running this:
> 
> stap -e 'probe syscall.clock_gettime, syscall.getuidr
>          { if (pid() == target()) printf("%s 0x%x in %s\n", name, uaddr(),
>                                          umodname(uaddr())) }' \
>      -c testsuite/vma_vdso-m64.exe
> 
> Which on x86_64 gives:
> 
> clock_gettime 0x7fff689ff9cf in vdso
> getuid 0x3c5c0ed819 in /usr/lib64/libc-2.15.so
> getuid 0x3c5c0bb007 in /usr/lib64/libc-2.15.so
> 
> Does that fail for all invocations on ARM?
> 
> If it is the vdso tracker then you want to take a look at
> runtime/vma.c _stp_vma_match_vdso will give some debug output
> with stap -DDEBUG_TASK_FINDER_VMA
> 
> Cheers,
> 
> Mark

Hi Mark,

Does this this test need the uprobes support for this test to work?

Below is the output for 2.6.42.3-2.fc15.armv7hl.tegra kernel which doesn't have uprobes support:

$ stap -e 'probe syscall.clock_gettime, syscall.getuid 
      { if (pid() == target()) printf("%s 0x%x in %s\n", name, uaddr(),
                                       umodname(uaddr())) }' -c testsuite/vma_vdsodefault.exe
clock_gettime 0x4010d6ec in <unknown>
getuid 0x40258e10 in <unknown>
getuid 0x40225f5c in <unknown>

-Will

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[Mark Wielaard]

On Mon, 2012-02-27 at 10:33 -0500, William Cohen wrote:
> On 02/25/2012 02:39 PM, Mark Wielaard wrote:
> 
> > A simpler test would be running this:
> > 
> > stap -e 'probe syscall.clock_gettime, syscall.getuidr
> >          { if (pid() == target()) printf("%s 0x%x in %s\n", name, uaddr(),
> >                                          umodname(uaddr())) }' \
> >      -c testsuite/vma_vdso-m64.exe
> > 
> > Which on x86_64 gives:
> > 
> > clock_gettime 0x7fff689ff9cf in vdso
> > getuid 0x3c5c0ed819 in /usr/lib64/libc-2.15.so
> > getuid 0x3c5c0bb007 in /usr/lib64/libc-2.15.so
> > 
> > Does that fail for all invocations on ARM?
> > 
> > If it is the vdso tracker then you want to take a look at
> > runtime/vma.c _stp_vma_match_vdso will give some debug output
> > with stap -DDEBUG_TASK_FINDER_VMA
> > 
> Does this this test need the uprobes support for this test to work?

Maybe... I am a little fuzzy on the details of when/how vma tracking is
triggered. Technically vma tracking doesn't need uprobes. But it might
be that vma tracking is only enabled/started if uprobes are active. You
might have to look with -DDEBUG_TASK_FINDER_VMA to see what is
happening.

> Below is the output for 2.6.42.3-2.fc15.armv7hl.tegra kernel which doesn't have uprobes support:
> 
> $ stap -e 'probe syscall.clock_gettime, syscall.getuid 
>       { if (pid() == target()) printf("%s 0x%x in %s\n", name, uaddr(),
>                                        umodname(uaddr())) }' -c testsuite/vma_vdsodefault.exe
> clock_gettime 0x4010d6ec in <unknown>
> getuid 0x40258e10 in <unknown>
> getuid 0x40225f5c in <unknown>

It is surprising none of these addresses map to a known vma map name. I
would expect at least some of them to come through libc and/or the
executable itself. You might want to extend the test scope to some other
(all?) syscalls and see which user address called into the kernel and
which umodname that corresponds to.

And maybe turn on -DDEBUG_TASK_FINDER_VMA to see which address ranges
are known to the stap runtime.

Cheers,

Mark

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[Mark Wielaard]

On Mon, Feb 27, 2012 at 05:10:08PM +0100, Mark Wielaard wrote:
> On Mon, 2012-02-27 at 10:33 -0500, William Cohen wrote:
> > Does this this test need the uprobes support for this test to work?
> 
> Maybe... I am a little fuzzy on the details of when/how vma tracking is
> triggered. Technically vma tracking doesn't need uprobes. But it might
> be that vma tracking is only enabled/started if uprobes are active. You
> might have to look with -DDEBUG_TASK_FINDER_VMA to see what is
> happening.

I poked a bit myself on an arm setup. The vma tracker doesn't need
uprobes, but it does need the task finder, and runtime.h says:

#if (defined(CONFIG_UTRACE) || defined(STAPCONF_UTRACE_VIA_TRACEPOINTS) \
     || defined(STAPCONF_UTRACE_VIA_FTRACE))
#define HAVE_TASK_FINDER
#endif

So you do need some form of UTRACE.
Unfortunately my setup doesn't have that.
Does your kernel?

Cheers,

Mark

Re: Recent review of SystemTap test results on ARM running Fedora 15 hard float.

[William Cohen]

On 02/27/2012 04:54 PM, Mark Wielaard wrote:
> On Mon, Feb 27, 2012 at 05:10:08PM +0100, Mark Wielaard wrote:
>> On Mon, 2012-02-27 at 10:33 -0500, William Cohen wrote:
>>> Does this this test need the uprobes support for this test to work?
>>
>> Maybe... I am a little fuzzy on the details of when/how vma tracking is
>> triggered. Technically vma tracking doesn't need uprobes. But it might
>> be that vma tracking is only enabled/started if uprobes are active. You
>> might have to look with -DDEBUG_TASK_FINDER_VMA to see what is
>> happening.
> 
> I poked a bit myself on an arm setup. The vma tracker doesn't need
> uprobes, but it does need the task finder, and runtime.h says:
> 
> #if (defined(CONFIG_UTRACE) || defined(STAPCONF_UTRACE_VIA_TRACEPOINTS) \
>      || defined(STAPCONF_UTRACE_VIA_FTRACE))
> #define HAVE_TASK_FINDER
> #endif
> 
> So you do need some form of UTRACE.
> Unfortunately my setup doesn't have that.
> Does your kernel?
> 
> Cheers,
> 
> Mark

Hi Mark,

My arm machine kernels don't have the UTRACE supoort either. I have the fedora 15 arm kernel (kernel-tegra-2.6.42.3-2.fc15.armv7hl) and the vanilla git tree kernel, 3.3.0-rc5+, on my machine.

-Will