* Summary of nightly tests 20060220
@ 2006-02-20 16:40 William Cohen
2006-02-20 19:52 ` Martin Hunt
0 siblings, 1 reply; 7+ messages in thread
From: William Cohen @ 2006-02-20 16:40 UTC (permalink / raw)
To: SystemTAP
The most severe problem is the FC i686 2.6.15-1.1831_FC4 kernel double
faults on one of the test systemtap.samples/poll_map.exp
One minor annoyance of the stap translator is that it prints
out the same error messages multiple times on the tests that fail.
FC4 i686 Linux 2.6.15-1.1831_FC4
(double fault in systemtap.samples/poll_map.exp)
FC5 i686 Linux 2.6.15-1.1948_FC5smp
Stap translator summary of failed tests
buildok-seven.stp:
Unable to find identifier $type in function find_pid
Unable to find identifier $task in function detach_pid
buildok-syscall.stp:
Unable to find identifier $timeout in function sys_poll (syscall2.stp)
Unable to find identifier $regs in function sys_rt_sigsuspend
(syscall2.stp)
Testsuite summary of failed tests
=== systemtap Summary ===
# of expected passes 129
FC5 x86_64 Linux 2.6.15-1.1948_FC5 (1 processor)
Stap translator summary of failed tests
buildok-seven.stp: unable to find identifier $task for function detach_pid
buildok-syscall.stp: unable to resolve function sys_mmap2
Testsuite summary of failed tests
=== systemtap Summary ===
# of expected passes 128
# of unsupported tests 1
RHEL4U3 i686 Linux 2.6.9-30.EL
Stap translator summary of failed tests
buildok-syscall.stp:
cannot resolve $regs used in tapset/syscalls2.stp (sys_rt_sigsuspend)
Testsuite summary of failed tests
=== systemtap Summary ===
# of expected passes 128
# of unsupported tests 1
RHEL4U3 x86_64 Linux 2.6.9-30.ELsmp
Stap translator summary of failed tests
buildok-syscall.stp: unable to resolve function sys_mmap2
Testsuite summary of failed tests
FAIL: ./systemtap.stress/current.stp startup (eof): test harness problem
=== systemtap Summary ===
# of expected passes 126
# of unexpected failures 1
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Summary of nightly tests 20060220
2006-02-20 16:40 Summary of nightly tests 20060220 William Cohen
@ 2006-02-20 19:52 ` Martin Hunt
2006-02-20 20:05 ` William Cohen
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Martin Hunt @ 2006-02-20 19:52 UTC (permalink / raw)
To: William Cohen; +Cc: systemtap
> FC5 i686 Linux 2.6.15-1.1948_FC5smp
>
> Stap translator summary of failed tests
> buildok-seven.stp:
> Unable to find identifier $type in function find_pid
> Unable to find identifier $task in function detach_pid
There don't appear to be significant changes in the sources to account
for this. That makes it a likely gcc bug.
Can someone write up the proper procedure to use to report these? I
think it should be on the web page.
Martin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Summary of nightly tests 20060220
2006-02-20 19:52 ` Martin Hunt
@ 2006-02-20 20:05 ` William Cohen
2006-02-20 20:36 ` Frank Ch. Eigler
2006-02-27 2:24 ` Suggestions for localizing problems with identifiers in SystemTap William Cohen
2 siblings, 0 replies; 7+ messages in thread
From: William Cohen @ 2006-02-20 20:05 UTC (permalink / raw)
To: Martin Hunt; +Cc: systemtap
Martin Hunt wrote:
>>FC5 i686 Linux 2.6.15-1.1948_FC5smp
>>
>>Stap translator summary of failed tests
>>buildok-seven.stp:
>> Unable to find identifier $type in function find_pid
>> Unable to find identifier $task in function detach_pid
>
>
> There don't appear to be significant changes in the sources to account
> for this. That makes it a likely gcc bug.
>
> Can someone write up the proper procedure to use to report these? I
> think it should be on the web page.
>
> Martin
>
>
The problem appears only in certain kernels. Possible causes for this
type of problem:
1) source code has changed so those variable/arguments don't exist
2) gcc tool chain not generating the proper debug information
3) stap is not finding them in the debugging information
Are there other possibilities or should this be finer grain resolution,
e.g. elfutils vs stap translator? The procedure should make it clear
which of the alternatives can be ruled out. Ideally, would like it to
point to to which one is causing the problem.
-Will
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Summary of nightly tests 20060220
2006-02-20 19:52 ` Martin Hunt
2006-02-20 20:05 ` William Cohen
@ 2006-02-20 20:36 ` Frank Ch. Eigler
2006-02-27 2:24 ` Suggestions for localizing problems with identifiers in SystemTap William Cohen
2 siblings, 0 replies; 7+ messages in thread
From: Frank Ch. Eigler @ 2006-02-20 20:36 UTC (permalink / raw)
To: systemtap
hunt wrote:
> > buildok-seven.stp:
> > Unable to find identifier $type in function find_pid
> > Unable to find identifier $task in function detach_pid
>
> There don't appear to be significant changes in the sources to
> account for this. That makes it a likely gcc bug. [...]
Yes, possibly. We will likely have a RH compiler person working on an
ongoing basis to fix such problems.
I can't find a message I remember writing to the mailing list about
the change I committed to buildok/seven.stp on 2006-02-07. So to
recap: It adds the "-u" stap flag to force all $target variables to be
expanded. This, while valuable to some extent in exposing debuginfo
problems, was meant to be a stopgap measure. To be considered
complete, the system calls tapset should eventually come with a test
case that exercises all the auxiliary / embedded-C functions, at least
at the buildok level. (This is what src/HACKING suggests.) At that
point, I would remove "-u", and let unresolved $target variables pass
quietly.
- FChE
^ permalink raw reply [flat|nested] 7+ messages in thread
* Suggestions for localizing problems with identifiers in SystemTap.
2006-02-20 19:52 ` Martin Hunt
2006-02-20 20:05 ` William Cohen
2006-02-20 20:36 ` Frank Ch. Eigler
@ 2006-02-27 2:24 ` William Cohen
2006-02-27 4:40 ` Roland McGrath
2 siblings, 1 reply; 7+ messages in thread
From: William Cohen @ 2006-02-27 2:24 UTC (permalink / raw)
To: Martin Hunt; +Cc: systemtap
Martin Hunt wrote:
>>FC5 i686 Linux 2.6.15-1.1948_FC5smp
>>
>>Stap translator summary of failed tests
>>buildok-seven.stp:
>> Unable to find identifier $type in function find_pid
>> Unable to find identifier $task in function detach_pid
>
>
> There don't appear to be significant changes in the sources to account
> for this. That makes it a likely gcc bug.
>
> Can someone write up the proper procedure to use to report these? I
> think it should be on the web page.
>
> Martin
I have jotted down some notes on how to track down these types of
problems with the identifiers. The notes are below. There are still lots
of holes.
-Will
Suggestions for localizing problems with identifiers in SystemTap.
Possible causes for this problems symbols in SystemTap scripts:
1) source code doesn't have function, variable, or argument
2) GCC tool chain not generating the proper debug information
3) system tap translator is unable them in the debugging information
-----
Source code doesn't have function, variable, or argument
This can be determined by examining the source code.
1) Find out which kernel is being used and location of source
code. You will need to have the cscope rpm installed
uname -r
get the matching source SRPM.
install srpm on the machine with:
rpm -Uvh kernel-<version>.src.rpm
cd into SPEC directory
rpmbuild -bp kernel-2.6.rpm
cd into BUILD/kernel-<version>/linux-<version> directory
make mrproper
cp /boot/config-`uname -r` .config; make oldconfig; make cscope
2) Find where the problem function is located and examine code for
variable or arument name. Ccsope is extremely useful for navigating
locating the functions.
Start up cscope and use the following cscope entries:
"Find this global definition:" or "Find this egrep pattern:"
Type in function name or pattern
If function shows up, hit return to look at function by hitting enter.
cscope will start an editor. Take a look for the problem symbol
if the function is "static" or "static inline",
the compiler could optimize these away if not called.
cscope can also show where the function is used
-----
GCC toolchain not generating the proper debug information
After establishing that the the symbol exists in source code,
determine whether the compiler is generating the proper debug
information. The program "readelf" in the binutils RPM can be used
the examine the debugging information in the kernel and modules.
readelf --debug-dump /usr/lib/debug/lib/modules/`uname -r`/vmlinux >
/tmp/debuginfo
This will generate a very large file /tmp/debuginfo that can be
examined. For example want to find out why $task is not being found
in function detach_pid for src/testsuite/buildok/seven.stp.
Looking for the section of the output in /tmp/debuginfo for the
detach_pid function we find the following section:
<1><3554ae>: Abbrev Number: 83 (DW_TAG_subprogram)
DW_AT_sibling : <35550b>
DW_AT_external : 1
DW_AT_name : (indirect string, offset: 0x1c783): detach_pid
DW_AT_decl_file : 1
DW_AT_decl_line : 193
DW_AT_prototyped : 1
DW_AT_low_pc : 0xffffffff80143458
DW_AT_high_pc : 0xffffffff80143499
DW_AT_frame_base : 0x8ee43 (location list)
<2><3554cf>: Abbrev Number: 76 (DW_TAG_formal_parameter)
DW_AT_name : (indirect string, offset: 0xba9d3): task
DW_AT_decl_file : 1
DW_AT_decl_line : 192
DW_AT_type : <35184d>
DW_AT_location : 0x8eea3 (location list)
<2><3554de>: Abbrev Number: 76 (DW_TAG_formal_parameter)
DW_AT_name : (indirect string, offset: 0x4fb9b): type
DW_AT_decl_file : 1
DW_AT_decl_line : 192
DW_AT_type : <35062a>
DW_AT_location : 0x8eec6 (location list)
<2><3554ed>: Abbrev Number: 72 (DW_TAG_variable)
DW_AT_name : tmp
DW_AT_decl_file : 1
DW_AT_decl_line : 194
DW_AT_type : <34e505>
DW_AT_location : 0x8ef0f (location list)
<2><3554fc>: Abbrev Number: 72 (DW_TAG_variable)
DW_AT_name : nr
DW_AT_decl_file : 1
DW_AT_decl_line : 194
DW_AT_type : <34e505>
DW_AT_location : 0x8ef45 (location list)
It is clear the "task" identifier is in the debugging information and
that it is a argument being passed into the function. The type of the
identifier can be determined by by searching for the "DW_AT_type", in
this case "><35184d>":
<1><35184d>: Abbrev Number: 7 (DW_TAG_pointer_type)
DW_AT_byte_size : 8
DW_AT_type : <350f16>
The chain can be followed searching for "><350f16>":
<1><350f16>: Abbrev Number: 6 (DW_TAG_typedef)
DW_AT_name : (indirect string, offset: 0x782ad): task_t
DW_AT_decl_file : 9
DW_AT_decl_line : 184
DW_AT_type : <34e8a0>
Searching for "><34e8a0>" yield task "task_struct" followed by its fields:
<1><34e8a0>: Abbrev Number: 15 (DW_TAG_structure_type)
DW_AT_sibling : <34f062>
DW_AT_name : (indirect string, offset: 0x15daa): task_struct
DW_AT_byte_size : 1888
DW_AT_decl_file : 10
DW_AT_decl_line : 13
<2><34e8ad>: Abbrev Number: 16 (DW_TAG_member)
DW_AT_name : (indirect string, offset: 0xb09cc): state
DW_AT_decl_file : 9
DW_AT_decl_line : 701
DW_AT_type : <34f60a>
DW_AT_data_member_location: 2 byte block: 23 0
(DW_OP_plus_uconst: 0)
<2><34e8bc>: Abbrev Number: 16 (DW_TAG_member)
DW_AT_name : (indirect string, offset: 0x8efe): thread_info
DW_AT_decl_file : 9
DW_AT_decl_line : 702
DW_AT_type : <35191d>
DW_AT_data_member_location: 2 byte block: 23 8
(DW_OP_plus_uconst: 8)
-----
SystemTap (stap) is not finding the identiers in the debugging information
After eliminating the source code and compiler, the SystemTap
translator is the likely suspect. The systemtap translator might be
able to find the identifier but is unable to process it. For example
src/testsuite/buildok/seve.stp instruments the detach_pid funtion in
the kernel and on the x86_64 produces the following error message:
semantic error: unresolved target-symbol expression: identifier '$task'
at
/home/wcohen/stap_testing_200602261550/src/testsuite/buildok/seven.stp:19:35
One can use one (or more) "-v" (verbose) options with the SystemTap
translator to get get a clearer picture of what the translator is
doing and getting some idea of why the symbol is not being resolved
correctly. The following command was used:
stap -p4 -vvvv ../src/testsuite/buildok/seven.stp
The previous command's output included:
pattern 'detach_pid' matches function 'detach_pid'
selected function detach_pid
finding prologue for 'detach_pid' entrypc=0xffffffff80143458
highpc=0xffffffff80143499
finding location for local 'task' near address ffffffff8014345f, module
bias 0
finding location for local 'type' near address ffffffff8014345f, module
bias 0
finding location for local 'task' near address ffffffff8014345f, module
bias 0
finding location for local 'type' near address ffffffff8014345f, module
bias 0
Thus, the 'task' local is found. However, the SystemTap translator
appears to be unable to handle it.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Suggestions for localizing problems with identifiers in SystemTap.
2006-02-27 2:24 ` Suggestions for localizing problems with identifiers in SystemTap William Cohen
@ 2006-02-27 4:40 ` Roland McGrath
2006-02-27 15:57 ` William Cohen
0 siblings, 1 reply; 7+ messages in thread
From: Roland McGrath @ 2006-02-27 4:40 UTC (permalink / raw)
To: systemtap
The place I would start is with the "loc2c-test" program. This is a simple
test program built during the systemtap build, but not installed. It uses
exactly the same infrastructure for finding and interpreting the debug
information to identify variables at requested source locations.
loc2c-test takes some standard options (see --help), which are the same as
you can use with e.g. eu-addr2line. Use the -k option to look at the
current kernel and modules, the same way the systemtap translator does.
(The other options make it easy to look at info for different kernels.)
The required argument is an address constant, giving the PC location of the
probe. The -v output from stap should tell you this.
./loc2c-test -k 0xc012c9b1
This yields:
kernel/pid.c (0x11)
current_stack_pointer [2e9735] long unsigned int
pid_hash [2e9752] hlist_head*[]
pidhash_shift [2e9763] int
pidmap_array [2e9784] pidmap_t[]
pidmap_lock [2e9795] spinlock_t
__kcrctab_find_task_by_pid_type[2e97a6] long unsigned int const
__kstrtab_find_task_by_pid_type[2e97c7] char[] const
__ksymtab_find_task_by_pid_type[2e97dd] kernel_symbol const
console_printk [2e97f9] int[]
mmu_cr4_features [2e9806] long unsigned int
xtime [2e9813] timespec
time_status [2e9820] int
time_maxerror [2e982d] long int
time_esterror [2e983a] long int
time_adjust [2e9847] long int
contig_page_data [2e9854] pglist_data
malloc_sizes [2e986d] cache_sizes[]
last_pid [2e987a] int
init_task [2e988c] task_struct
irq_desc [2e98aa] irq_desc_t[]
irq_vector [2e98c7] u8[]
io_apic_irqs [2e98d4] long unsigned int
per_cpu__rcu_data [2e98e1] rcu_data
per_cpu__rcu_bh_data [2e98ee] rcu_data
rcu_ctrlblk [2e98fb] rcu_ctrlblk
rcu_bh_ctrlblk [2e9908] rcu_ctrlblk
dcache_lock [2e9915] spinlock_t
acpi_noirq [2e9922] int
acpi_disabled [2e992f] int
acpi_ht [2e993c] int
acpi_pci_disabled [2e9949] int
skip_ioapic_setup [2e9956] int
zone_table [2e996e] zone*[]
mem_map [2e997c] page*
swapper_space [2e998a] address_space
per_cpu__cpu_gdt_table [2e99a8] desc_struct[]
default_ldt [2e99c0] desc_struct[]
dma_spin_lock [2e99cd] spinlock_t
nr_kernel_pages [2e99da] long unsigned int
pid_max [2e99e7] int
pid_max_min [2e99f9] int
pid_max_max [2e9a0b] int
__crc_find_task_by_pid_type [2e9a1d] <no type>*
detach_pid (0x2e): 0xc012c9b1 (kernel/pid.c:193) .. 0xc012c9ed (kernel/pid.c:208)
task [2e9436] task_t*
type [2e9445] pid_type
tmp [2e9454] int
nr [2e9463] int
The shows the DWARF scopes that contain the PC address for the probe point,
and the variable names defined in each. Here you see the file (global)
scope, and inside it the "detach_pid" function scope. If a variable is not
shown here, then it wasn't found in the DWARF info. You can use readelf or
eu-readelf to investigate something you find missing here.
Thanks,
Roland
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Suggestions for localizing problems with identifiers in SystemTap.
2006-02-27 4:40 ` Roland McGrath
@ 2006-02-27 15:57 ` William Cohen
0 siblings, 0 replies; 7+ messages in thread
From: William Cohen @ 2006-02-27 15:57 UTC (permalink / raw)
To: Roland McGrath; +Cc: systemtap
Roland McGrath wrote:
> The place I would start is with the "loc2c-test" program. This is a simple
> test program built during the systemtap build, but not installed. It uses
> exactly the same infrastructure for finding and interpreting the debug
> information to identify variables at requested source locations.
>
> loc2c-test takes some standard options (see --help), which are the same as
> you can use with e.g. eu-addr2line. Use the -k option to look at the
> current kernel and modules, the same way the systemtap translator does.
> (The other options make it easy to look at info for different kernels.)
> The required argument is an address constant, giving the PC location of the
> probe. The -v output from stap should tell you this.
>
> ./loc2c-test -k 0xc012c9b1
Hi Roland,
Thanks. for the explanation on how to use loc2c-test. It is pretty easy
to get the address of the "function of interest" in the kernel with
something like the following:
nm /usr/lib/debug/lib/modules/`uname -r`/vmlinux \
|grep " detach_pid$" |awk '{print "0x" $1}'
How would one get the addresses for inlined functions and functions in
modules?
-Will
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2006-02-27 15:57 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-02-20 16:40 Summary of nightly tests 20060220 William Cohen
2006-02-20 19:52 ` Martin Hunt
2006-02-20 20:05 ` William Cohen
2006-02-20 20:36 ` Frank Ch. Eigler
2006-02-27 2:24 ` Suggestions for localizing problems with identifiers in SystemTap William Cohen
2006-02-27 4:40 ` Roland McGrath
2006-02-27 15:57 ` William Cohen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).