Test suite results for ARM with uprobes

Test suite results for ARM with uprobes

[Wade Farnsworth]

Hi all,

I ran the systemtap test suite on a beagleboard with uprobes support 
enabled.  However, I'm running into several errors.

My configuration is as follows:
* systemtap git rev f52d32a9f57d228627ee08e39f0bbcf3f3faae20.
* kernel 2.6.37.6
* gcc is 4.5.1

The full logs may be found at:

http://dl.dropbox.com/u/40714612/stap-logs-beagleboard-20111206.tar.gz

I realize this is quite a large list to work through, but I would be 
very appreciative if anyone could shed any light on them.  If there's 
anything obvious that I'm missing I'd be glad to hear it.  Otherwise, 
I'll continue digging at them.

First, four tests are causing hangs, panics or other kernel errors.  I 
had to disable these tests in order to proceed further:

* systemtap.base/pr10854.exp results in the following:
INIT: Id "S" respawning too fast: disabled for 5 minutes
Kernel panic - not syncing: Attempted to kill init!

* systemtap.base/itrace.exp will hang in run_ls_1_sec at the line "catch 
{wait -i $exe_id}".  It looks like the ls process never exits, so we 
just spin on this line.  If I comment out this line, I hit the kernel 
BUG in linux/kernel/exit.c:forget_original_parent() followed by a NULL 
pointer dereference in do_exit().

* systemtap.examples/check.exp gets an "Unable to handle kernel paging 
request at virtual address" error

* systemtap.unprivileged/unprivileged_myproc.exp hits the BUG at 
runtime/uprobes2/uprobes.c:uprobe_free_task()

I'm guessing that there may be some bugs in the ARM uprobes 
implementation, but I haven't yet tracked these down any further.

Once these tests have been disabled, the rest of the test suite will 
complete.  Here are some of the failures that I'm not sure about and 
some relevant notes:

FAIL: cast-scope-m32-O
FAIL: cast-scope-m32-O2
[WARNING: Can't parse SDT_V3 operand '[fp,': identifier '$arg1' at 
/home/root/stuff/systemtap/testsuite/systemtap.base/cast-scope.stp:15:32]

FAIL: cxxclass-m32
FAIL: cxxclass-m32-O
FAIL: cxxclass-m32-O2
[Got "semantic error: unable to find local 'arg1' near pc 0x842c  in 
<unknown> 
/home/root/stuff/systemtap/testsuite/systemtap.base/cxxclass.cxx ( 
(alternatives: $i $inst): identifier '$arg1' at 
/home/root/stuff/systemtap/testsuite/systemtap.base/cxxclass.stp:13:24"]

FAIL: debugpath-good (eof) [This one puzzles me.  I've built with 
CONFIG_DEBUG_INFO enabled and the build directory exists in 
/lib/modules/`uname -r`/build.  Is there something else that I need to 
do to get systemtap access to the debug info?]

FAIL: flightrec2 (log file size (5 != 3 + 3)) [stat: cannot stat 
`flightlog.out.1': No such file or directory]

FAIL: global_end (11) [Looks like the queue stats being taken in 
global_end2.stp is not occurring]

FAIL: gtod (0) [events marked "kern" are occurring before events marked 
"appl"]

FAIL: inlinedvars-m32-O
FAIL: inlinedvars-m32-O2
[line 1: expected "call (22,84)"
Got "  (84,22)"]

FAIL: library sdt_misc * (0 != 15)
FAIL: library sdt_misc *libsdt* (0 != 15)
FAIL: library sdt_misc ./libsdt.so (0 != 15)
FAIL: library printf --ldd (0) (0 != 4)
[Looks like systemtap needs to be made aware of the ARM library loader]

FAIL: OVERLOAD2 no expected error

FAIL: probefunc:.statement.(0xaddr).absolute shutdown (eof) [systemtap 
appears to be unable to translate the address back into the 
corresponding symbol.  Could be related to the debugpath failure?]

FAIL: systemtap.base/process_by_cmd.stp -c ./process_by_cmd [Oddly, 
systemtap is picking up a function return before the process starts. 
Possibly it's coming from libc?]

FAIL: sdt -O2  uprobe
FAIL: sdt c89  uprobe
FAIL: sdt c99  uprobe
FAIL: sdt c99 -pedantic uprobe
FAIL: sdt gnu99  uprobe
FAIL: sdt gnu99 -pedantic uprobe
FAIL: sdt c++98  uprobe
FAIL: sdt c++98 -pedantic uprobe
FAIL: sdt gnu++98  uprobe
FAIL: sdt gnu++98 -pedantic uprobe
FAIL: sdt c++0x  uprobe
FAIL: sdt c++0x -pedantic uprobe
FAIL: sdt gnu++0x  uprobe
FAIL: sdt gnu++0x -pedantic uprobe
[semantic error: unable to find local 'arg1' near pc 0x8408  in  call1 
/home/root/stuff/systemtap/testsuite/systemtap.base/sdt.c ( 
(alternatives: $a): identifier '$arg1' at 
/home/root/stuff/systemtap/testsuite/systemtap.base/sdt.stp:8:18.]

FAIL: sdt_va_args base
FAIL: sdt_va_args c89
FAIL: sdt_va_args c99
FAIL: sdt_va_args gnu99
FAIL: sdt_va_args c++98
FAIL: sdt_va_args gnu++98
FAIL: sdt_va_args c++0x
FAIL: sdt_va_args gnu++0x
[similar to the sdt failures above]

FAIL: compiling sdt.c c89 -pedantic uprobe 
[/home/root/stuff/systemtap/testsuite/systemtap.base/sdt.c:67:3: error: 
string length '518' is greater than the length '509' ISO C90 compilers 
are required to support]

ERROR: tcl error sourcing 
/home/root/stuff/systemtap/testsuite/systemtap.base/sdt_misc.exp [ERROR: 
child process exited abnormally]

FAIL: stmt_rel line numbers [semantic error: multiple addresses for...]

FAIL: 32_BIT_UTRACE_SYSCALL_ARGS startup (eof) [Warning: child process 
exited with signal 4 (Illegal instruction)]

FAIL: vta-test-m32-O
FAIL: vta-test-m32-O2
[semantic error: failed to retrieve location attribute for local 'a' 
(dieoffset: 0x181): identifier '$a' at 
/home/root/stuff/systemtap/testsuite/systemtap.base/vta-test.stp:2:27]

FAIL: dtrace_clone2 compilation
FAIL: dtrace_clone4 compilation
[WARNING: Can't parse SDT_V3 operand 'r3': identifier '$arg1' at 
/home/root/stuff/systemtap/testsuite/systemtap.clone/dtrace_clone.stp:6:47]

FAIL: dtrace_fork_exec2 compilation
FAIL: dtrace_fork_exec4 compilation
FAIL: dtrace_vfork_exec2 compilation
FAIL: dtrace_vfork_exec4 compilation
[Fail similarly to dtrace_clone]

FAIL: backtrace of yyy_func2 (0)
FAIL: print_stack of yyy_func2 (0)
FAIL: backtrace of yyy_func3 (0)
FAIL: print_stack of yyy_func3 (0)
FAIL: backtrace of yyy_func4 (0)
FAIL: print_stack of yyy_func4 (0)
FAIL: print_stack didn't find systemtap_test_module1 (0)
FAIL: print_stack didn't find [kernel] (0)
FAIL: function arguments: unexpected timeout
FAIL: all pid tests - unexpected EOF
FAIL: function arguments -- numeric: compilation failed
FAIL: function arguments -- numeric --kelf --ignore-dwarf: compilation 
failed
[semantic error: no match while resolving probe point 
module("systemtap-test-module2").function("yyy_int")]

FAIL: usymbols m32
FAIL: usymbols m32-O
FAIL: usymbols m32-O2
[line 2: expected "handler: lib_handler (.+/libusymbols-m32.so)", Got 
"handler: 0x400b450c (/home/root/stuff/test/libusymbols-m32.so)"]

FAIL: buildok/pretty.stp [Looks like I need to rerun this with the 
systemtap debug info installed]

FAIL: buildok/process_test.stp [semantic error: unable to find local 
'signr' near pc 0xc005f5b4  in  handle_signal...]

FAIL: buildok/scheduler-all-probes.stp [semantic error: no match while 
resolving probe point kernel.function("__switch_to")]

FAIL: buildok/seventeen.stp [semantic error: unable to find local 
'nfs_program' near pc 0xc01bc714  in  nfs_fsync_dir]

FAIL: buildok/syscalls-arch-detailed.stp
[semantic error: probe point mismatch at position 1]

FAIL: semok/config_number.stp [semantic error: probe point mismatch at 
position 0]

FAIL: semok/mangled.stp
FAIL: semok/pretty.stp
[similar to buildok/pretty.stp]

FAIL: semok/twentyseven.stp [semantic error: no match while resolving 
probe point module("no_such_module").function("no_such_function")]

FAIL: systemtap.stress/current.stp compilation [similar to 
buildok/process_test.stp]

FAIL: 32-bit access nd_syscall
FAIL: 32-bit acct nd_syscall
FAIL: 32-bit alarm nd_syscall
FAIL: 32-bit chmod nd_syscall
FAIL: 32-bit clock nd_syscall
FAIL: 32-bit dir nd_syscall
FAIL: 32-bit forkwait nd_syscall
FAIL: 32-bit futimes nd_syscall
FAIL: 32-bit itimer nd_syscall
FAIL: 32-bit link nd_syscall
FAIL: 32-bit mmap nd_syscall
FAIL: 32-bit mount nd_syscall
FAIL: 32-bit net1 nd_syscall
FAIL: 32-bit openclose nd_syscall
FAIL: 32-bit readwrite nd_syscall
FAIL: 32-bit rt_signal nd_syscall
FAIL: 32-bit select nd_syscall
FAIL: 32-bit sendfile nd_syscall
FAIL: 32-bit signal nd_syscall
FAIL: 32-bit stat nd_syscall
FAIL: 32-bit statfs nd_syscall
FAIL: 32-bit swap nd_syscall
FAIL: 32-bit sync nd_syscall
FAIL: 32-bit timer nd_syscall
FAIL: 32-bit trunc nd_syscall
FAIL: 32-bit uid nd_syscall
FAIL: 32-bit umask nd_syscall
FAIL: 32-bit unlink nd_syscall

FAIL: 32-bit alarm syscall
FAIL: 32-bit stat syscall

If anyone has any clues about the above failures, please let me know!

Thanks!

-Wade Farnsworth

Re: Test suite results for ARM with uprobes

[Mark Wielaard]

Hi Wade,

On Tue, 2011-12-06 at 12:45 -0700, Wade Farnsworth wrote:
> I ran the systemtap test suite on a beagleboard with uprobes support 
> enabled.

Thanks!

>   However, I'm running into several errors.
> 
> My configuration is as follows:
> * systemtap git rev f52d32a9f57d228627ee08e39f0bbcf3f3faae20.
> * kernel 2.6.37.6
> * gcc is 4.5.1

I don't yet have a setup that includes arm uprobes, sorry. But I do have
results for the non-uprobe side of things based on almost the same git
checkout, kernel 2.6.40.4-6.fc15.armv7hl.tegra and gcc (GCC) 4.6.1
20110908 (Red Hat 4.6.1-9).

> The full logs may be found at:
> 
> http://dl.dropbox.com/u/40714612/stap-logs-beagleboard-20111206.tar.gz
> 
> I realize this is quite a large list to work through, but I would be 
> very appreciative if anyone could shed any light on them.  If there's 
> anything obvious that I'm missing I'd be glad to hear it.  Otherwise, 
> I'll continue digging at them.

My results can be found at:
http://web.elastic.org/~dejazilla/viewsummary.php?summary=%3D%27%
3C20111202161840.2C07549CD7%40springer.wildebeest.org%3E%27
Lets compare at least the non-uprobes dependent tests.

> First, four tests are causing hangs, panics or other kernel errors.

In general I found that arm kernels pre-3.0 (2.6.40 in fedora speak)
were somewhat unstable. There were lots of kprobe cleanups in 3.0.

> I 
> had to disable these tests in order to proceed further:
> 
> * systemtap.base/pr10854.exp results in the following:
> INIT: Id "S" respawning too fast: disabled for 5 minutes
> Kernel panic - not syncing: Attempted to kill init!

This one takes 60 seconds, but does PASS for me.

> * systemtap.base/itrace.exp will hang in run_ls_1_sec at the line "catch 
> {wait -i $exe_id}".  It looks like the ls process never exits, so we 
> just spin on this line.  If I comment out this line, I hit the kernel 
> BUG in linux/kernel/exit.c:forget_original_parent() followed by a NULL 
> pointer dereference in do_exit().

This one is UNTESTED for me because no kernel utrace support found.

> * systemtap.examples/check.exp gets an "Unable to handle kernel paging 
> request at virtual address" error

This is a lot of tests. It is a run of all the example scripts we ship
with systemtap. It takes 52 minutes to run this whole test on my setup.
Although there are 24 UNTESTED runs in this test, all other 162 PASS.
There are no FAILs here.

> * systemtap.unprivileged/unprivileged_myproc.exp hits the BUG at 
> runtime/uprobes2/uprobes.c:uprobe_free_task()

I am unable to run this test because I don't have uprobes setup here
yet.

> I'm guessing that there may be some bugs in the ARM uprobes 
> implementation, but I haven't yet tracked these down any further.
> 
> Once these tests have been disabled, the rest of the test suite will 
> complete.  Here are some of the failures that I'm not sure about and 
> some relevant notes:
> 
> FAIL: cast-scope-m32-O
> FAIL: cast-scope-m32-O2
> [WARNING: Can't parse SDT_V3 operand '[fp,': identifier '$arg1' at 
> /home/root/stuff/systemtap/testsuite/systemtap.base/cast-scope.stp:15:32]
> FAIL: cxxclass-m32
> FAIL: cxxclass-m32-O
> FAIL: cxxclass-m32-O2
> [Got "semantic error: unable to find local 'arg1' near pc 0x842c  in 
> <unknown> 
> /home/root/stuff/systemtap/testsuite/systemtap.base/cxxclass.cxx ( 
> (alternatives: $i $inst): identifier '$arg1' at 
> /home/root/stuff/systemtap/testsuite/systemtap.base/cxxclass.stp:13:24"]

Although I am unable to actually run these tests, this looks like a
genuine bug in the SDT probes parser. I filed a bug report:
http://sourceware.org/bugzilla/show_bug.cgi?id=13474

> FAIL: debugpath-good (eof) [This one puzzles me.  I've built with 
> CONFIG_DEBUG_INFO enabled and the build directory exists in 
> /lib/modules/`uname -r`/build.  Is there something else that I need to 
> do to get systemtap access to the debug info?]

It fails because:
spawn env SYSTEMTAP_DEBUGINFO_PATH=/lib/modules/2.6.37.6-yocto-standard+/build s
tap -e probe kernel.function("vfs_read") {} -wp2
semantic error: missing arm kernel/module debuginfo under '/lib/modules/2.6.37.6
-yocto-standard+/build' while resolving probe point kernel.function("vfs_read")
Pass 2: analysis failed.  Try again with another '--vp 01' option.
FAIL: debugpath-good (eof)

It PASSes for me with:
spawn env SYSTEMTAP_DEBUGINFO_PATH=/usr/lib/debug stap -e probe kernel.function(
"vfs_read") {} -wp2
# probes
kernel.function("vfs_read@fs/read_write.c:306") /* pc=_stext+0x1445e0 */ /* <- k
ernel.function("vfs_read") */
PASS: debugpath-good

The debugpath.exp testcase does:

# Guess where debuginfo is installed
if [file isdirectory /usr/lib/debug] {
  set debuginfo_path "/usr/lib/debug"
} elseif [file isdirectory /lib/modules/$uname/build] {
  set debuginfo_path "/lib/modules/$uname/build"
} else {
  set debuginfo_path "/lib/modules/$uname"
}

So, I guess it is guessing wrongly?

> FAIL: flightrec2 (log file size (5 != 3 + 3)) [stat: cannot stat 
> `flightlog.out.1': No such file or directory]

Also FAILs for me:
FAIL: flightrec2 (log file size (5 != 3 + 3))

> FAIL: global_end (11) [Looks like the queue stats being taken in 
> global_end2.stp is not occurring]

Also FAILs for me:
FAIL: global_end (11)

> FAIL: gtod (0) [events marked "kern" are occurring before events marked 
> "appl"]

Also FAILs for me:
FAIL: gtod (0)
I suspect this testcase is a little flaky, I have seen it fail on other
setups too sometimes.

> FAIL: inlinedvars-m32-O
> FAIL: inlinedvars-m32-O2
> [line 1: expected "call (22,84)"
> Got "  (84,22)"]

That is interesting, it switches the numbers around?
It obviously doesn't run for me. It would be interesting to see the
debuginfo debug-dump of the created binary, the disassembly of function
"m" and the generated stap script C source code. Could you create a bug
report with that info in it?

> FAIL: library sdt_misc * (0 != 15)
> FAIL: library sdt_misc *libsdt* (0 != 15)
> FAIL: library sdt_misc ./libsdt.so (0 != 15)
> FAIL: library printf --ldd (0) (0 != 4)
> [Looks like systemtap needs to be made aware of the ARM library loader]

Yes, I think so.

> FAIL: OVERLOAD2 no expected error

Same here:
FAIL: OVERLOAD2 no expected error

> FAIL: probefunc:.statement.(0xaddr).absolute shutdown (eof) [systemtap 
> appears to be unable to translate the address back into the 
> corresponding symbol.  Could be related to the debugpath failure?]

Indeed, it does not seem to translate the address back into the
corresponding symbol, but I don't think that it is because of the
debugpath failure. I don't immediately know what it is though. This and
all other probefunc tests PASS for me BTW (the answer is
scheduler_tick).

> FAIL: systemtap.base/process_by_cmd.stp -c ./process_by_cmd [Oddly, 
> systemtap is picking up a function return before the process starts. 
> Possibly it's coming from libc?]

I am unable to run this test.

> FAIL: sdt -O2  uprobe
> FAIL: sdt c89  uprobe
> FAIL: sdt c99  uprobe
> FAIL: sdt c99 -pedantic uprobe
> FAIL: sdt gnu99  uprobe
> FAIL: sdt gnu99 -pedantic uprobe
> FAIL: sdt c++98  uprobe
> FAIL: sdt c++98 -pedantic uprobe
> FAIL: sdt gnu++98  uprobe
> FAIL: sdt gnu++98 -pedantic uprobe
> FAIL: sdt c++0x  uprobe
> FAIL: sdt c++0x -pedantic uprobe
> FAIL: sdt gnu++0x  uprobe
> FAIL: sdt gnu++0x -pedantic uprobe
> [semantic error: unable to find local 'arg1' near pc 0x8408  in  call1 
> /home/root/stuff/systemtap/testsuite/systemtap.base/sdt.c ( 
> (alternatives: $a): identifier '$arg1' at 
> /home/root/stuff/systemtap/testsuite/systemtap.base/sdt.stp:8:18.]
> 
> FAIL: sdt_va_args base
> FAIL: sdt_va_args c89
> FAIL: sdt_va_args c99
> FAIL: sdt_va_args gnu99
> FAIL: sdt_va_args c++98
> FAIL: sdt_va_args gnu++98
> FAIL: sdt_va_args c++0x
> FAIL: sdt_va_args gnu++0x
> [similar to the sdt failures above]

I am unable to run these tests, but it would be interesting to see
debuginfo dump, disassembly and generated stap script C source for
these.

> FAIL: compiling sdt.c c89 -pedantic uprobe 
> [/home/root/stuff/systemtap/testsuite/systemtap.base/sdt.c:67:3: error: 
> string length '518' is greater than the length '509' ISO C90 compilers 
> are required to support]

Known issue with older GCC. Fixed in newer GCC versions.

> ERROR: tcl error sourcing 
> /home/root/stuff/systemtap/testsuite/systemtap.base/sdt_misc.exp [ERROR: 
> child process exited abnormally]

I cannot run this test, and don't have clue what is going wrong here.

> FAIL: stmt_rel line numbers [semantic error: multiple addresses for...]

Bleah, that keeps popping up. It is very dependent on the compiler
generated code. Wish we could do something better here. It happens to
PASS on my kernel build though.

> FAIL: 32_BIT_UTRACE_SYSCALL_ARGS startup (eof) [Warning: child process 
> exited with signal 4 (Illegal instruction)]

I don't have a UTRACE enable kernel here.

> FAIL: vta-test-m32-O
> FAIL: vta-test-m32-O2
> [semantic error: failed to retrieve location attribute for local 'a' 
> (dieoffset: 0x181): identifier '$a' at 
> /home/root/stuff/systemtap/testsuite/systemtap.base/vta-test.stp:2:27]

I cannot run this tests. It would be interesting to see the debuginfo
dump for the generated binary.

> FAIL: dtrace_clone2 compilation
> FAIL: dtrace_clone4 compilation
> [WARNING: Can't parse SDT_V3 operand 'r3': identifier '$arg1' at 
> /home/root/stuff/systemtap/testsuite/systemtap.clone/dtrace_clone.stp:6:47]
>
> FAIL: dtrace_fork_exec2 compilation
> FAIL: dtrace_fork_exec4 compilation
> FAIL: dtrace_vfork_exec2 compilation
> FAIL: dtrace_vfork_exec4 compilation
> [Fail similarly to dtrace_clone]

Same as for the other SDT parser bug above. I'll filed:
http://sourceware.org/bugzilla/show_bug.cgi?id=13475

> FAIL: backtrace of yyy_func2 (0)
> FAIL: print_stack of yyy_func2 (0)
> FAIL: backtrace of yyy_func3 (0)
> FAIL: print_stack of yyy_func3 (0)
> FAIL: backtrace of yyy_func4 (0)
> FAIL: print_stack of yyy_func4 (0)
> FAIL: print_stack didn't find systemtap_test_module1 (0)
> FAIL: print_stack didn't find [kernel] (0)
> FAIL: function arguments: unexpected timeout
> FAIL: all pid tests - unexpected EOF
> FAIL: function arguments -- numeric: compilation failed
> FAIL: function arguments -- numeric --kelf --ignore-dwarf: compilation 
> failed
> [semantic error: no match while resolving probe point 
> module("systemtap-test-module2").function("yyy_int")]

These all PASS for me (this was what I worked on last week).
I don't know why, but I know Will was seeing something similar:
http://www.sourceware.org/bugzilla/show_bug.cgi?id=13022
For some reason, it seems fine on my setup, which is why I haven't
really looked into this yet.

> FAIL: usymbols m32
> FAIL: usymbols m32-O
> FAIL: usymbols m32-O2
> [line 2: expected "handler: lib_handler (.+/libusymbols-m32.so)", Got 
> "handler: 0x400b450c (/home/root/stuff/test/libusymbols-m32.so)"]

Symbol lookup again, but this time user space. I have these UNTESTED.

> FAIL: buildok/pretty.stp [Looks like I need to rerun this with the 
> systemtap debug info installed]

This one passes for me:
PASS: buildok/pretty.stp
But only because the testcase skips UTRACE dependent tests.

It looks like the testcase might be using the wrong stap:
WARNING: cannot find module /usr/bin/stap debuginfo: No DWARF
information found
I don't have systemtap itself installed on my machine.

Yep, it probably does, see buildok/pretty.stp has:
  probe process("stap").function("parse_cmdline") {
...
hmmm, need to think how to fix that.

> FAIL: buildok/process_test.stp [semantic error: unable to find local 
> 'signr' near pc 0xc005f5b4  in  handle_signal...]

Same here.
Looks like tapset/signal.stp handle_signal is wrong for ARM.

> FAIL: buildok/scheduler-all-probes.stp [semantic error: no match while 
> resolving probe point kernel.function("__switch_to")]

Same here.
Looks like there is no __switch_to() in ARM kernels.
Note that tapset/scheduler.stp has a couple of:
%( arch != "x86_64" && arch != "ia64" %?
        kernel.function("__switch_to")
%:
        kernel.function("context_switch")
%)
So we need to adjust that for ARM.

> FAIL: buildok/seventeen.stp [semantic error: unable to find local 
> 'nfs_program' near pc 0xc01bc714  in  nfs_fsync_dir]

This one does PASS for me. We should probably compare generated
debuginfo for our kernels.

> FAIL: buildok/syscalls-arch-detailed.stp
> [semantic error: probe point mismatch at position 1]

Same here. We are missing syscall.altstack on ARM it seems.

> FAIL: semok/config_number.stp [semantic error: probe point mismatch at 
> position 0]

This one PASSes for me.
What is CONFIG_NR_CPU set to for your kernel?
It is set to 2 for me.

> FAIL: semok/mangled.stp
> FAIL: semok/pretty.stp
> [similar to buildok/pretty.stp]
>
> FAIL: semok/twentyseven.stp [semantic error: no match while resolving 
> probe point module("no_such_module").function("no_such_function")]
> 
> FAIL: systemtap.stress/current.stp compilation [similar to 
> buildok/process_test.stp]
> 
> FAIL: 32-bit access nd_syscall
> FAIL: 32-bit acct nd_syscall
> FAIL: 32-bit alarm nd_syscall
> FAIL: 32-bit chmod nd_syscall
> FAIL: 32-bit clock nd_syscall
> FAIL: 32-bit dir nd_syscall
> FAIL: 32-bit forkwait nd_syscall
> FAIL: 32-bit futimes nd_syscall
> FAIL: 32-bit itimer nd_syscall
> FAIL: 32-bit link nd_syscall
> FAIL: 32-bit mmap nd_syscall
> FAIL: 32-bit mount nd_syscall
> FAIL: 32-bit net1 nd_syscall
> FAIL: 32-bit openclose nd_syscall
> FAIL: 32-bit readwrite nd_syscall
> FAIL: 32-bit rt_signal nd_syscall
> FAIL: 32-bit select nd_syscall
> FAIL: 32-bit sendfile nd_syscall
> FAIL: 32-bit signal nd_syscall
> FAIL: 32-bit stat nd_syscall
> FAIL: 32-bit statfs nd_syscall
> FAIL: 32-bit swap nd_syscall
> FAIL: 32-bit sync nd_syscall
> FAIL: 32-bit timer nd_syscall
> FAIL: 32-bit trunc nd_syscall
> FAIL: 32-bit uid nd_syscall
> FAIL: 32-bit umask nd_syscall
> FAIL: 32-bit unlink nd_syscall

We are only able to get arguments of these syscalls in registers,
whenever some argument spills onto the stack we ERROR out. This is the
code in tapset/arm/registers.stp _stp_arg().

> FAIL: 32-bit alarm syscall
> FAIL: 32-bit stat syscall

Both also fail for me, but I haven't investigated yet.

Cheers,

Mark

Re: Test suite results for ARM with uprobes

[Wade Farnsworth]

Hi Mark,

Thanks for your analysis!

Mark Wielaard wrote:
 > On Tue, 2011-12-06 at 12:45 -0700, Wade Farnsworth wrote:
[...]
>> First, four tests are causing hangs, panics or other kernel errors.
>
> In general I found that arm kernels pre-3.0 (2.6.40 in fedora speak)
> were somewhat unstable. There were lots of kprobe cleanups in 3.0.
>

Ah, yes.  You did mention that previously.  I'll see about retesting 
with something more recent.

[...]
>> FAIL: debugpath-good (eof) [This one puzzles me.  I've built with
>> CONFIG_DEBUG_INFO enabled and the build directory exists in
>> /lib/modules/`uname -r`/build.  Is there something else that I need to
>> do to get systemtap access to the debug info?]
>
> It fails because:
> spawn env SYSTEMTAP_DEBUGINFO_PATH=/lib/modules/2.6.37.6-yocto-standard+/build s
> tap -e probe kernel.function("vfs_read") {} -wp2
> semantic error: missing arm kernel/module debuginfo under '/lib/modules/2.6.37.6
> -yocto-standard+/build' while resolving probe point kernel.function("vfs_read")
> Pass 2: analysis failed.  Try again with another '--vp 01' option.
> FAIL: debugpath-good (eof)
>
> It PASSes for me with:
> spawn env SYSTEMTAP_DEBUGINFO_PATH=/usr/lib/debug stap -e probe kernel.function(
> "vfs_read") {} -wp2
> # probes
> kernel.function("vfs_read@fs/read_write.c:306") /* pc=_stext+0x1445e0 */ /*<- k
> ernel.function("vfs_read") */
> PASS: debugpath-good
>
> The debugpath.exp testcase does:
>
> # Guess where debuginfo is installed
> if [file isdirectory /usr/lib/debug] {
>    set debuginfo_path "/usr/lib/debug"
> } elseif [file isdirectory /lib/modules/$uname/build] {
>    set debuginfo_path "/lib/modules/$uname/build"
> } else {
>    set debuginfo_path "/lib/modules/$uname"
> }
>
> So, I guess it is guessing wrongly?
>

Hmm, I do have the debuginfo-enabled vmlinux in 
/lib/modules/2.6.37.6-yocto-standard+/build, which stap seems to 
otherwise be able to find.  For some reason it seems to only occur when 
SYSTEMTAP_DEBUGINFO_PATH is used.  I'll do some more digging.

[...]
>
>> FAIL: inlinedvars-m32-O
>> FAIL: inlinedvars-m32-O2
>> [line 1: expected "call (22,84)"
>> Got "  (84,22)"]
>
> That is interesting, it switches the numbers around?
> It obviously doesn't run for me. It would be interesting to see the
> debuginfo debug-dump of the created binary, the disassembly of function
> "m" and the generated stap script C source code. Could you create a bug
> report with that info in it?

Done.  http://sourceware.org/bugzilla/show_bug.cgi?id=13485

[...]
>
>> FAIL: sdt -O2  uprobe
>> FAIL: sdt c89  uprobe
>> FAIL: sdt c99  uprobe
>> FAIL: sdt c99 -pedantic uprobe
>> FAIL: sdt gnu99  uprobe
>> FAIL: sdt gnu99 -pedantic uprobe
>> FAIL: sdt c++98  uprobe
>> FAIL: sdt c++98 -pedantic uprobe
>> FAIL: sdt gnu++98  uprobe
>> FAIL: sdt gnu++98 -pedantic uprobe
>> FAIL: sdt c++0x  uprobe
>> FAIL: sdt c++0x -pedantic uprobe
>> FAIL: sdt gnu++0x  uprobe
>> FAIL: sdt gnu++0x -pedantic uprobe
>> [semantic error: unable to find local 'arg1' near pc 0x8408  in  call1
>> /home/root/stuff/systemtap/testsuite/systemtap.base/sdt.c (
>> (alternatives: $a): identifier '$arg1' at
>> /home/root/stuff/systemtap/testsuite/systemtap.base/sdt.stp:8:18.]
>>
>> FAIL: sdt_va_args base
>> FAIL: sdt_va_args c89
>> FAIL: sdt_va_args c99
>> FAIL: sdt_va_args gnu99
>> FAIL: sdt_va_args c++98
>> FAIL: sdt_va_args gnu++98
>> FAIL: sdt_va_args c++0x
>> FAIL: sdt_va_args gnu++0x
>> [similar to the sdt failures above]
>
> I am unable to run these tests, but it would be interesting to see
> debuginfo dump, disassembly and generated stap script C source for
> these.

The failure occurs before the C source is generated I have uploaded the 
other info to:
http://dl.dropbox.com/u/40714612/stap-debug-files.tar.gz

[...]
>> FAIL: vta-test-m32-O
>> FAIL: vta-test-m32-O2
>> [semantic error: failed to retrieve location attribute for local 'a'
>> (dieoffset: 0x181): identifier '$a' at
>> /home/root/stuff/systemtap/testsuite/systemtap.base/vta-test.stp:2:27]
>
> I cannot run this tests. It would be interesting to see the debuginfo
> dump for the generated binary.

Also in the tarball on dropbox.

[...]
>
>> FAIL: buildok/seventeen.stp [semantic error: unable to find local
>> 'nfs_program' near pc 0xc01bc714  in  nfs_fsync_dir]
>
> This one does PASS for me. We should probably compare generated
> debuginfo for our kernels.
>

Kernel debuginfo is also in the tarball.

[...]
>> FAIL: semok/config_number.stp [semantic error: probe point mismatch at
>> position 0]
>
> This one PASSes for me.
> What is CONFIG_NR_CPU set to for your kernel?
> It is set to 2 for me.
>

CONFIG_SMP is not set, so CONFIG_NR_CPU doesn't exist in my config file. 
  Maybe stap treats that as being equal to zero?

I'll keep poking at the other failures.  Thanks again!

Regards,

-Wade Farnsworth

Re: Test suite results for ARM with uprobes

[Mark Wielaard]

On Thu, 2011-12-08 at 08:20 -0700, Wade Farnsworth wrote:
> >> FAIL: semok/config_number.stp [semantic error: probe point mismatch
> at
> >> position 0]
> >
> > This one PASSes for me.
> > What is CONFIG_NR_CPU set to for your kernel?
> > It is set to 2 for me.
> >
> 
> CONFIG_SMP is not set, so CONFIG_NR_CPU doesn't exist in my config
> file. 
>   Maybe stap treats that as being equal to zero?

Yep it does. Although I am not sure it is meant to. It uses strtoll to
convert the empty string "", which returns zero, then it checks if errno
is set and it isn't, even though the man page says "The  implementation
may also set errno to EINVAL in case no conversion was performed (no
digits seen, and 0 returned)."

I don't know what is preferred. I could change the code to barf on an
empty string in a CONFIG_ numerical comparison, then we would need to
change the testcase to pick some other CONFIG_ option which is always
set. I think this is the most sane, but it is a slight change in how
things worked before (I think by accident).

Or we could accept that numerical comparisons of CONFIG_ options which
aren't set compare to zero and then we change the testcase to also
accept that CONFIG_NR_CPU == 0.

Opinions?

Thanks,

Mark

Re: Test suite results for ARM with uprobes

[Mark Wielaard]

[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]

On Wed, Dec 14, 2011 at 04:32:03PM +0100, Mark Wielaard wrote:
> On Thu, 2011-12-08 at 08:20 -0700, Wade Farnsworth wrote:
> > >> FAIL: semok/config_number.stp [semantic error: probe point mismatch
> > at
> > >> position 0]
> > >
> > > This one PASSes for me.
> > > What is CONFIG_NR_CPU set to for your kernel?
> > > It is set to 2 for me.
> > >
> > 
> > CONFIG_SMP is not set, so CONFIG_NR_CPU doesn't exist in my config
> > file. 
> >   Maybe stap treats that as being equal to zero?
> 
> Yep it does. Although I am not sure it is meant to. It uses strtoll to
> convert the empty string "", which returns zero, then it checks if errno
> is set and it isn't, even though the man page says "The  implementation
> may also set errno to EINVAL in case no conversion was performed (no
> digits seen, and 0 returned)."
> 
> I don't know what is preferred. I could change the code to barf on an
> empty string in a CONFIG_ numerical comparison, then we would need to
> change the testcase to pick some other CONFIG_ option which is always
> set. I think this is the most sane, but it is a slight change in how
> things worked before (I think by accident).
> 
> Or we could accept that numerical comparisons of CONFIG_ options which
> aren't set compare to zero and then we change the testcase to also
> accept that CONFIG_NR_CPU == 0.
> 
> Opinions?

Even though nobody gave their opinion, I did change mine.
I have added an explicit test for the current behaviour.
Non set CONFIG_options should compare equal to the empty string
and/or zero. That also should make the test work on your setup.

Even though I still think that is kind of ugly, it seems better
to keep current behaviour that people/scripts might already depend
on. And now if we do change it, we at least have a test that points
out the change in behaviour.

Cheers,

Mark

[-- Attachment #2: config_nr_test.patch --]
[-- Type: text/plain, Size: 887 bytes --]

commit 0a51c2a2dea8a2cafa0e642087547f48bfe0fa08
Author: Mark Wielaard <mjw@redhat.com>
Date:   Fri Dec 16 19:14:40 2011 +0100

    Check that unset CONFIG options compare equal the empty string and/or zero.

diff --git a/testsuite/semok/config_number.stp b/testsuite/semok/config_number.stp
index b384c3e..ba3bde7 100755
--- a/testsuite/semok/config_number.stp
+++ b/testsuite/semok/config_number.stp
@@ -1,14 +1,15 @@
 #! stap -p2
 
 # check that number comparisons work in CONFIG checks
+# Note that unset CONFIG options compare equal the empty string and/or zero
 probe
   %( CONFIG_NR_CPUS == 13 %?
     noprobe
   %:
-    %( CONFIG_NR_CPUS < 1 %?
+    %( CONFIG_NR_CPUS != "" && CONFIG_NR_CPUS < 1 %?
       nonoprobe
     %:
-      %( CONFIG_NR_CPUS >= 0 %?
+      %( CONFIG_NR_CPUS >= 0 && CONFIG_NO_SUCH_FOOBAR_CONFIG_OPTION == 0 %?
         begin
       %:
         nononoprobe