Testsuite run on ppc64

Testsuite run on ppc64

[Hien Nguyen]

Here's the testsuite result for ppc64. There are 4 failed tests listed 
below and my comments:

1)  FAIL: ./systemtap.base/kmodule.stp compilation  -
probably because the test could not find stap_run

2) Running ./systemtap.samples/symbols.exp ...
FAIL: symbols (1)

If I run the "stap symbols", I could count 11 line, looks like the test 
looks for 11 count but some how it still failed
# stap symbols.stp
 0x0
 0x80000000
 0xc0000000
 0xe0000000
 0xf0000000
 0xffffffff
 0xffffffff00000000 : .xfrm6_state_fini+0x2ffffffeffc7a65c/0x0 [ipv6]
 0xffffffff80000000 : .xfrm6_state_fini+0x2fffffff7fc7a65c/0x0 [ipv6]
 0xffffffff80120000 : .xfrm6_state_fini+0x2fffffff7fd9a65c/0x0 [ipv6]
 0xffffffff88000000 : .xfrm6_state_fini+0x2fffffff87c7a65c/0x0 [ipv6]
 0xffffffffffffffff : .xfrm6_state_fini+0x2fffffffffc7a65b/0x0 [ipv6]

3) Running ./systemtap.samples/syscalls1.exp ...
FAIL: syscalls-count (24)

The test looks for 250-300 system calls, but I run the below and it 
seems to give the correct value
# stap -p2 syscalls.stp | wc -l
290


4) Running ./systemtap.samples/syscalls2.exp ...
FAIL: syscalls-run (4)

The  looks for 100-150 count, run the below and it seems to give the 
correct value.
# stap syscalls.stp | wc -l
101


------ Test suite result starts here -----

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file 
for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for 
target.
Using ./config/unix.exp as tool-and-target-specific interface file.
Running ./systemtap/systemtap-run.exp ...
Running ./systemtap.base/add.exp ...
Running ./systemtap.base/and.exp ...
Running ./systemtap.base/dec.exp ...
Running ./systemtap.base/div.exp ...
Running ./systemtap.base/div0.exp ...
Running ./systemtap.base/equal.exp ...
Running ./systemtap.base/finloop2.exp ...
Running ./systemtap.base/gt.exp ...
Running ./systemtap.base/gte.exp ...
Running ./systemtap.base/if.exp ...
Running ./systemtap.base/inc.exp ...
Running ./systemtap.base/kfunct.exp ...
Running ./systemtap.base/kmodule.exp ...
FAIL: ./systemtap.base/kmodule.stp compilation
Running ./systemtap.base/logical_and.exp ...
Running ./systemtap.base/logical_not.exp ...
Running ./systemtap.base/logical_or.exp ...
Running ./systemtap.base/lshift.exp ...
Running ./systemtap.base/lt.exp ...
Running ./systemtap.base/lte.exp ...
Running ./systemtap.base/mod.exp ...
Running ./systemtap.base/mult.exp ...
Running ./systemtap.base/ne.exp ...
Running ./systemtap.base/neg.exp ...
Running ./systemtap.base/not.exp ...
Running ./systemtap.base/or.exp ...
Running ./systemtap.base/rshift.exp ...
Running ./systemtap.base/simple.exp ...
Running ./systemtap.base/subtract.exp ...
Running ./systemtap.base/tri.exp ...
Running ./systemtap.base/xor.exp ...
Running ./systemtap.samples/arith.exp ...
Running ./systemtap.samples/arith_limits.exp ...
Running ./systemtap.samples/control_limits.exp ...
Running ./systemtap.samples/pfaults.exp ...
Running ./systemtap.samples/primes.exp ...
Running ./systemtap.samples/profile.exp ...
Running ./systemtap.samples/symbols.exp ...
FAIL: symbols (1)
Running ./systemtap.samples/syscalls1.exp ...
FAIL: syscalls-count (24)
Running ./systemtap.samples/syscalls2.exp ...
FAIL: syscalls-run (4)
Running ./systemtap.samples/sysopen.exp ...
Running ./systemtap.samples/transport.exp ...

                ===  Summary ===

# of expected passes            101
# of unexpected failures        4

Re: Testsuite run on ppc64

[William Cohen]

Hien, Thanks for running the the tests on PPC64. The systemtap.log won't 
give the details on the compilation failure, stap hides thoes details. 
Could you do the following?

For systemtap.base/kmodule.stp:

1) stap -V -k  ./systemtap.base/kmodule.stp compilation >& error.txt
2) post the error.txt and a tarball of the generated kernel module

For ./systemtap.samples/symbols.exp it looks like it only matched one 
line. Maybe modify the ./systemtap.samples/symbols.exp like the 
following to find out which line it is matching on. It looks like the 
regular expression needs to be corrected.

set test "symbols"

spawn stap -g ./$subdir/symbols.stp
set ok 0
expect {
     -timeout 30
     -re { 0x[a-f0-9]+.*\r} { incr ok; pass "$test matches"; exp_continue; }
     timeout { fail "$test (timeout)" }
     eof { }
}

It is odd that the results for the automated run of syscall1.stp and 
syscall2.stp differ from running by hand.

-Will

Hien Nguyen wrote:
> Here's the testsuite result for ppc64. There are 4 failed tests listed 
> below and my comments:
> 
> 1)  FAIL: ./systemtap.base/kmodule.stp compilation  -
> probably because the test could not find stap_run
> 
> 2) Running ./systemtap.samples/symbols.exp ...
> FAIL: symbols (1)
> 
> If I run the "stap symbols", I could count 11 line, looks like the test 
> looks for 11 count but some how it still failed
> # stap symbols.stp
> 0x0
> 0x80000000
> 0xc0000000
> 0xe0000000
> 0xf0000000
> 0xffffffff
> 0xffffffff00000000 : .xfrm6_state_fini+0x2ffffffeffc7a65c/0x0 [ipv6]
> 0xffffffff80000000 : .xfrm6_state_fini+0x2fffffff7fc7a65c/0x0 [ipv6]
> 0xffffffff80120000 : .xfrm6_state_fini+0x2fffffff7fd9a65c/0x0 [ipv6]
> 0xffffffff88000000 : .xfrm6_state_fini+0x2fffffff87c7a65c/0x0 [ipv6]
> 0xffffffffffffffff : .xfrm6_state_fini+0x2fffffffffc7a65b/0x0 [ipv6]
> 
> 3) Running ./systemtap.samples/syscalls1.exp ...
> FAIL: syscalls-count (24)
> 
> The test looks for 250-300 system calls, but I run the below and it 
> seems to give the correct value
> # stap -p2 syscalls.stp | wc -l
> 290
> 
> 
> 4) Running ./systemtap.samples/syscalls2.exp ...
> FAIL: syscalls-run (4)
> 
> The  looks for 100-150 count, run the below and it seems to give the 
> correct value.
> # stap syscalls.stp | wc -l
> 101
> 
> 
> ------ Test suite result starts here -----
> 
> Running target unix
> Using /usr/share/dejagnu/baseboards/unix.exp as board description file 
> for target.
> Using /usr/share/dejagnu/config/unix.exp as generic interface file for 
> target.
> Using ./config/unix.exp as tool-and-target-specific interface file.
> Running ./systemtap/systemtap-run.exp ...
> Running ./systemtap.base/add.exp ...
> Running ./systemtap.base/and.exp ...
> Running ./systemtap.base/dec.exp ...
> Running ./systemtap.base/div.exp ...
> Running ./systemtap.base/div0.exp ...
> Running ./systemtap.base/equal.exp ...
> Running ./systemtap.base/finloop2.exp ...
> Running ./systemtap.base/gt.exp ...
> Running ./systemtap.base/gte.exp ...
> Running ./systemtap.base/if.exp ...
> Running ./systemtap.base/inc.exp ...
> Running ./systemtap.base/kfunct.exp ...
> Running ./systemtap.base/kmodule.exp ...
> FAIL: ./systemtap.base/kmodule.stp compilation
> Running ./systemtap.base/logical_and.exp ...
> Running ./systemtap.base/logical_not.exp ...
> Running ./systemtap.base/logical_or.exp ...
> Running ./systemtap.base/lshift.exp ...
> Running ./systemtap.base/lt.exp ...
> Running ./systemtap.base/lte.exp ...
> Running ./systemtap.base/mod.exp ...
> Running ./systemtap.base/mult.exp ...
> Running ./systemtap.base/ne.exp ...
> Running ./systemtap.base/neg.exp ...
> Running ./systemtap.base/not.exp ...
> Running ./systemtap.base/or.exp ...
> Running ./systemtap.base/rshift.exp ...
> Running ./systemtap.base/simple.exp ...
> Running ./systemtap.base/subtract.exp ...
> Running ./systemtap.base/tri.exp ...
> Running ./systemtap.base/xor.exp ...
> Running ./systemtap.samples/arith.exp ...
> Running ./systemtap.samples/arith_limits.exp ...
> Running ./systemtap.samples/control_limits.exp ...
> Running ./systemtap.samples/pfaults.exp ...
> Running ./systemtap.samples/primes.exp ...
> Running ./systemtap.samples/profile.exp ...
> Running ./systemtap.samples/symbols.exp ...
> FAIL: symbols (1)
> Running ./systemtap.samples/syscalls1.exp ...
> FAIL: syscalls-count (24)
> Running ./systemtap.samples/syscalls2.exp ...
> FAIL: syscalls-run (4)
> Running ./systemtap.samples/sysopen.exp ...
> Running ./systemtap.samples/transport.exp ...
> 
>                ===  Summary ===
> 
> # of expected passes            101
> # of unexpected failures        4
> 

Re: Testsuite run on ppc64

[Frank Ch. Eigler]

wcohen wrote:

> [...] For ./systemtap.samples/symbols.exp it looks like it only
> matched one line. Maybe modify the ./systemtap.samples/symbols.exp
> like the following to find out which line it is matching on. It
> looks like the regular expression needs to be corrected.  [...]

There may be a systemic error in the dejagnu regexps, where multiple
lines of output are generated by systemtap or the probe, and are
intended to be matched individually.  The "-re "SOMETHING.*\r"
idiom may need to become something like "-re SOMETHING[^\r]*\r".

- FChE

Re: Testsuite run on ppc64

[Hien Nguyen]

Frank Ch. Eigler wrote:

>There may be a systemic error in the dejagnu regexps, where multiple
>lines of output are generated by systemtap or the probe, and are
>intended to be matched individually.  The "-re "SOMETHING.*\r"
>idiom may need to become something like "-re SOMETHING[^\r]*\r".
>
>- FChE
>
>  
>
Bingo :-)

Re: Testsuite run on ppc64

[Hien Nguyen]

William Cohen wrote:

> Hien, Thanks for running the the tests on PPC64. The systemtap.log 
> won't give the details on the compilation failure, stap hides thoes 
> details. Could you do the following?
>
> For systemtap.base/kmodule.stp:
>
> 1) stap -V -k  ./systemtap.base/kmodule.stp compilation >& error.txt

# stap -v -k ./systemtap.base/kmodule.stp
Created temporary directory "/tmp/stapLIUsfb"
Searched '/usr/local/share/systemtap/tapset/2.6.9-22.EL/*.stp', match 
count 0
Searched '/usr/local/share/systemtap/tapset/2.6.9/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/2.6/*.stp', match count 0
Searched '/usr/local/share/systemtap/tapset/*.stp', match count 9
Pass 1: parsed user script and 9 library script(s).
parsed 'ext3_sync_file' -> func 'ext3_sync_file'
pattern 'ext3' matches module 'ext3'
focused on module 'ext3' = [7fffffffffffffff-80000000000324f8, bias 0]
WARNING: cannot find module ext3 debuginfo: No such file or directory
semantic error: no match for probe point
         while: resolving probe point 
module("ext3").function("ext3_sync_file")
Pass 2: analyzed user script.  2 probe(s), 10 function(s), 1 global(s).
Pass 2: analysis failed.  Try again with '-v' (verbose) option.
Keeping temporary directory "/tmp/stapLIUsfb"

I am not sure why it cound not find the debug info for ext3.
# rpm -qa | grep kernel
kernel-doc-2.6.9-22.EL
kernel-debuginfo-2.6.9-22.EL
kernel-utils-2.4-13.1.69
kernel-utils-2.4-13.1.69
kernel-devel-2.6.9-22.EL
kernel-2.6.9-22.EL

> 2) post the error.txt and a tarball of the generated kernel module
>
> For ./systemtap.samples/symbols.exp it looks like it only matched one 
> line. Maybe modify the ./systemtap.samples/symbols.exp like the 
> following to find out which line it is matching on. It looks like the 
> regular expression needs to be corrected.
>
> set test "symbols"
>
> spawn stap -g ./$subdir/symbols.stp
> set ok 0
> expect {
>     -timeout 30
>     -re { 0x[a-f0-9]+.*\r} { incr ok; pass "$test matches"; 
> exp_continue; }
>     timeout { fail "$test (timeout)" }
>     eof { }
> }

I get this test passed with Frank suggestion.

>
> It is odd that the results for the automated run of syscall1.stp and 
> syscall2.stp differ from running by hand.
>
> -Will
>

Re: Testsuite run on ppc64

[William Cohen]

Hien Nguyen wrote:
> Frank Ch. Eigler wrote:
> 
>> There may be a systemic error in the dejagnu regexps, where multiple
>> lines of output are generated by systemtap or the probe, and are
>> intended to be matched individually.  The "-re "SOMETHING.*\r"
>> idiom may need to become something like "-re SOMETHING[^\r]*\r".
>>
>> - FChE
>>
>>  
>>
> Bingo :-)

I just checked in changes based on Franks suggestions. For symbol.exp, 
syscall[1,2].exp. Those should provide better results.

-Will

Re: Testsuite run on ppc64

[Hien Nguyen]

William Cohen wrote:

> Hien Nguyen wrote:
>
>> Frank Ch. Eigler wrote:
>>
>>> There may be a systemic error in the dejagnu regexps, where multiple
>>> lines of output are generated by systemtap or the probe, and are
>>> intended to be matched individually.  The "-re "SOMETHING.*\r"
>>> idiom may need to become something like "-re SOMETHING[^\r]*\r".
>>>
>>> - FChE
>>>
>>>  
>>>
>> Bingo :-)
>
>
> I just checked in changes based on Franks suggestions. For symbol.exp, 
> syscall[1,2].exp. Those should provide better results.
>
> -Will
>
Yes, all tests passed except for the kmodule test . Look under 
/usr/lib/debug/lib/modules/2.6.9-22.EL, and yes the ext3 debuginfo is there.

#find . -name "ext3*"
./kernel/fs/ext3
./kernel/fs/ext3/ext3.ko.debug

Re: Testsuite run on ppc64

[William Cohen]

Hien Nguyen wrote:
> William Cohen wrote:
> 
>> Hien Nguyen wrote:
>>
>>> Frank Ch. Eigler wrote:
>>>
>>>> There may be a systemic error in the dejagnu regexps, where multiple
>>>> lines of output are generated by systemtap or the probe, and are
>>>> intended to be matched individually.  The "-re "SOMETHING.*\r"
>>>> idiom may need to become something like "-re SOMETHING[^\r]*\r".
>>>>
>>>> - FChE
>>>>
>>>>  
>>>>
>>> Bingo :-)
>>
>>
>>
>> I just checked in changes based on Franks suggestions. For symbol.exp, 
>> syscall[1,2].exp. Those should provide better results.
>>
>> -Will
>>
> Yes, all tests passed except for the kmodule test . Look under 
> /usr/lib/debug/lib/modules/2.6.9-22.EL, and yes the ext3 debuginfo is 
> there.
> 
> #find . -name "ext3*"
> ./kernel/fs/ext3
> ./kernel/fs/ext3/ext3.ko.debug
> 

Hien,

The revised tests work. Good.

Narrowing down the problem with kmodule.stp. Here is a list of possible 
failure modes and the first one looks likely:

1) The debuginfo information for the module is not correct or not available.
	I see the following error message in the output of "stap -V":

WARNING: cannot find module ext3 debuginfo: No such file or directory

	Check to see that kernel-debuginfo provides "ext3.ko.debug"

	rpm -qs kernel-debuginfo | grep "ext3.ko.debug"

	Does the following list the appropriate symbol(ext3_sync_file)?

nm /usr/lib/debug/lib/modules/`uname -r`/kernel/fs/ext3/ext3.ko.debug


2) Elfutils is not reading the information correctly.
	Which version of elfutils are you using? 0.116?
	x86_64 had problems with modules with earlier elfutils.
	Make sure that you have the elfutils-0.116 installed

3) Translator is missing the symbol. Could this be a symbol naming 
problem, e.g. compiler for ppc64 generates different labels.


-Will