* Summary of nightly tests 20060220 @ 2006-02-20 16:40 William Cohen 2006-02-20 19:52 ` Martin Hunt 0 siblings, 1 reply; 7+ messages in thread From: William Cohen @ 2006-02-20 16:40 UTC (permalink / raw) To: SystemTAP The most severe problem is the FC i686 2.6.15-1.1831_FC4 kernel double faults on one of the test systemtap.samples/poll_map.exp One minor annoyance of the stap translator is that it prints out the same error messages multiple times on the tests that fail. FC4 i686 Linux 2.6.15-1.1831_FC4 (double fault in systemtap.samples/poll_map.exp) FC5 i686 Linux 2.6.15-1.1948_FC5smp Stap translator summary of failed tests buildok-seven.stp: Unable to find identifier $type in function find_pid Unable to find identifier $task in function detach_pid buildok-syscall.stp: Unable to find identifier $timeout in function sys_poll (syscall2.stp) Unable to find identifier $regs in function sys_rt_sigsuspend (syscall2.stp) Testsuite summary of failed tests === systemtap Summary === # of expected passes 129 FC5 x86_64 Linux 2.6.15-1.1948_FC5 (1 processor) Stap translator summary of failed tests buildok-seven.stp: unable to find identifier $task for function detach_pid buildok-syscall.stp: unable to resolve function sys_mmap2 Testsuite summary of failed tests === systemtap Summary === # of expected passes 128 # of unsupported tests 1 RHEL4U3 i686 Linux 2.6.9-30.EL Stap translator summary of failed tests buildok-syscall.stp: cannot resolve $regs used in tapset/syscalls2.stp (sys_rt_sigsuspend) Testsuite summary of failed tests === systemtap Summary === # of expected passes 128 # of unsupported tests 1 RHEL4U3 x86_64 Linux 2.6.9-30.ELsmp Stap translator summary of failed tests buildok-syscall.stp: unable to resolve function sys_mmap2 Testsuite summary of failed tests FAIL: ./systemtap.stress/current.stp startup (eof): test harness problem === systemtap Summary === # of expected passes 126 # of unexpected failures 1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Summary of nightly tests 20060220 2006-02-20 16:40 Summary of nightly tests 20060220 William Cohen @ 2006-02-20 19:52 ` Martin Hunt 2006-02-20 20:05 ` William Cohen ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Martin Hunt @ 2006-02-20 19:52 UTC (permalink / raw) To: William Cohen; +Cc: systemtap > FC5 i686 Linux 2.6.15-1.1948_FC5smp > > Stap translator summary of failed tests > buildok-seven.stp: > Unable to find identifier $type in function find_pid > Unable to find identifier $task in function detach_pid There don't appear to be significant changes in the sources to account for this. That makes it a likely gcc bug. Can someone write up the proper procedure to use to report these? I think it should be on the web page. Martin ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Summary of nightly tests 20060220 2006-02-20 19:52 ` Martin Hunt @ 2006-02-20 20:05 ` William Cohen 2006-02-20 20:36 ` Frank Ch. Eigler 2006-02-27 2:24 ` Suggestions for localizing problems with identifiers in SystemTap William Cohen 2 siblings, 0 replies; 7+ messages in thread From: William Cohen @ 2006-02-20 20:05 UTC (permalink / raw) To: Martin Hunt; +Cc: systemtap Martin Hunt wrote: >>FC5 i686 Linux 2.6.15-1.1948_FC5smp >> >>Stap translator summary of failed tests >>buildok-seven.stp: >> Unable to find identifier $type in function find_pid >> Unable to find identifier $task in function detach_pid > > > There don't appear to be significant changes in the sources to account > for this. That makes it a likely gcc bug. > > Can someone write up the proper procedure to use to report these? I > think it should be on the web page. > > Martin > > The problem appears only in certain kernels. Possible causes for this type of problem: 1) source code has changed so those variable/arguments don't exist 2) gcc tool chain not generating the proper debug information 3) stap is not finding them in the debugging information Are there other possibilities or should this be finer grain resolution, e.g. elfutils vs stap translator? The procedure should make it clear which of the alternatives can be ruled out. Ideally, would like it to point to to which one is causing the problem. -Will ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Summary of nightly tests 20060220 2006-02-20 19:52 ` Martin Hunt 2006-02-20 20:05 ` William Cohen @ 2006-02-20 20:36 ` Frank Ch. Eigler 2006-02-27 2:24 ` Suggestions for localizing problems with identifiers in SystemTap William Cohen 2 siblings, 0 replies; 7+ messages in thread From: Frank Ch. Eigler @ 2006-02-20 20:36 UTC (permalink / raw) To: systemtap hunt wrote: > > buildok-seven.stp: > > Unable to find identifier $type in function find_pid > > Unable to find identifier $task in function detach_pid > > There don't appear to be significant changes in the sources to > account for this. That makes it a likely gcc bug. [...] Yes, possibly. We will likely have a RH compiler person working on an ongoing basis to fix such problems. I can't find a message I remember writing to the mailing list about the change I committed to buildok/seven.stp on 2006-02-07. So to recap: It adds the "-u" stap flag to force all $target variables to be expanded. This, while valuable to some extent in exposing debuginfo problems, was meant to be a stopgap measure. To be considered complete, the system calls tapset should eventually come with a test case that exercises all the auxiliary / embedded-C functions, at least at the buildok level. (This is what src/HACKING suggests.) At that point, I would remove "-u", and let unresolved $target variables pass quietly. - FChE ^ permalink raw reply [flat|nested] 7+ messages in thread
* Suggestions for localizing problems with identifiers in SystemTap. 2006-02-20 19:52 ` Martin Hunt 2006-02-20 20:05 ` William Cohen 2006-02-20 20:36 ` Frank Ch. Eigler @ 2006-02-27 2:24 ` William Cohen 2006-02-27 4:40 ` Roland McGrath 2 siblings, 1 reply; 7+ messages in thread From: William Cohen @ 2006-02-27 2:24 UTC (permalink / raw) To: Martin Hunt; +Cc: systemtap Martin Hunt wrote: >>FC5 i686 Linux 2.6.15-1.1948_FC5smp >> >>Stap translator summary of failed tests >>buildok-seven.stp: >> Unable to find identifier $type in function find_pid >> Unable to find identifier $task in function detach_pid > > > There don't appear to be significant changes in the sources to account > for this. That makes it a likely gcc bug. > > Can someone write up the proper procedure to use to report these? I > think it should be on the web page. > > Martin I have jotted down some notes on how to track down these types of problems with the identifiers. The notes are below. There are still lots of holes. -Will Suggestions for localizing problems with identifiers in SystemTap. Possible causes for this problems symbols in SystemTap scripts: 1) source code doesn't have function, variable, or argument 2) GCC tool chain not generating the proper debug information 3) system tap translator is unable them in the debugging information ----- Source code doesn't have function, variable, or argument This can be determined by examining the source code. 1) Find out which kernel is being used and location of source code. You will need to have the cscope rpm installed uname -r get the matching source SRPM. install srpm on the machine with: rpm -Uvh kernel-<version>.src.rpm cd into SPEC directory rpmbuild -bp kernel-2.6.rpm cd into BUILD/kernel-<version>/linux-<version> directory make mrproper cp /boot/config-`uname -r` .config; make oldconfig; make cscope 2) Find where the problem function is located and examine code for variable or arument name. Ccsope is extremely useful for navigating locating the functions. Start up cscope and use the following cscope entries: "Find this global definition:" or "Find this egrep pattern:" Type in function name or pattern If function shows up, hit return to look at function by hitting enter. cscope will start an editor. Take a look for the problem symbol if the function is "static" or "static inline", the compiler could optimize these away if not called. cscope can also show where the function is used ----- GCC toolchain not generating the proper debug information After establishing that the the symbol exists in source code, determine whether the compiler is generating the proper debug information. The program "readelf" in the binutils RPM can be used the examine the debugging information in the kernel and modules. readelf --debug-dump /usr/lib/debug/lib/modules/`uname -r`/vmlinux > /tmp/debuginfo This will generate a very large file /tmp/debuginfo that can be examined. For example want to find out why $task is not being found in function detach_pid for src/testsuite/buildok/seven.stp. Looking for the section of the output in /tmp/debuginfo for the detach_pid function we find the following section: <1><3554ae>: Abbrev Number: 83 (DW_TAG_subprogram) DW_AT_sibling : <35550b> DW_AT_external : 1 DW_AT_name : (indirect string, offset: 0x1c783): detach_pid DW_AT_decl_file : 1 DW_AT_decl_line : 193 DW_AT_prototyped : 1 DW_AT_low_pc : 0xffffffff80143458 DW_AT_high_pc : 0xffffffff80143499 DW_AT_frame_base : 0x8ee43 (location list) <2><3554cf>: Abbrev Number: 76 (DW_TAG_formal_parameter) DW_AT_name : (indirect string, offset: 0xba9d3): task DW_AT_decl_file : 1 DW_AT_decl_line : 192 DW_AT_type : <35184d> DW_AT_location : 0x8eea3 (location list) <2><3554de>: Abbrev Number: 76 (DW_TAG_formal_parameter) DW_AT_name : (indirect string, offset: 0x4fb9b): type DW_AT_decl_file : 1 DW_AT_decl_line : 192 DW_AT_type : <35062a> DW_AT_location : 0x8eec6 (location list) <2><3554ed>: Abbrev Number: 72 (DW_TAG_variable) DW_AT_name : tmp DW_AT_decl_file : 1 DW_AT_decl_line : 194 DW_AT_type : <34e505> DW_AT_location : 0x8ef0f (location list) <2><3554fc>: Abbrev Number: 72 (DW_TAG_variable) DW_AT_name : nr DW_AT_decl_file : 1 DW_AT_decl_line : 194 DW_AT_type : <34e505> DW_AT_location : 0x8ef45 (location list) It is clear the "task" identifier is in the debugging information and that it is a argument being passed into the function. The type of the identifier can be determined by by searching for the "DW_AT_type", in this case "><35184d>": <1><35184d>: Abbrev Number: 7 (DW_TAG_pointer_type) DW_AT_byte_size : 8 DW_AT_type : <350f16> The chain can be followed searching for "><350f16>": <1><350f16>: Abbrev Number: 6 (DW_TAG_typedef) DW_AT_name : (indirect string, offset: 0x782ad): task_t DW_AT_decl_file : 9 DW_AT_decl_line : 184 DW_AT_type : <34e8a0> Searching for "><34e8a0>" yield task "task_struct" followed by its fields: <1><34e8a0>: Abbrev Number: 15 (DW_TAG_structure_type) DW_AT_sibling : <34f062> DW_AT_name : (indirect string, offset: 0x15daa): task_struct DW_AT_byte_size : 1888 DW_AT_decl_file : 10 DW_AT_decl_line : 13 <2><34e8ad>: Abbrev Number: 16 (DW_TAG_member) DW_AT_name : (indirect string, offset: 0xb09cc): state DW_AT_decl_file : 9 DW_AT_decl_line : 701 DW_AT_type : <34f60a> DW_AT_data_member_location: 2 byte block: 23 0 (DW_OP_plus_uconst: 0) <2><34e8bc>: Abbrev Number: 16 (DW_TAG_member) DW_AT_name : (indirect string, offset: 0x8efe): thread_info DW_AT_decl_file : 9 DW_AT_decl_line : 702 DW_AT_type : <35191d> DW_AT_data_member_location: 2 byte block: 23 8 (DW_OP_plus_uconst: 8) ----- SystemTap (stap) is not finding the identiers in the debugging information After eliminating the source code and compiler, the SystemTap translator is the likely suspect. The systemtap translator might be able to find the identifier but is unable to process it. For example src/testsuite/buildok/seve.stp instruments the detach_pid funtion in the kernel and on the x86_64 produces the following error message: semantic error: unresolved target-symbol expression: identifier '$task' at /home/wcohen/stap_testing_200602261550/src/testsuite/buildok/seven.stp:19:35 One can use one (or more) "-v" (verbose) options with the SystemTap translator to get get a clearer picture of what the translator is doing and getting some idea of why the symbol is not being resolved correctly. The following command was used: stap -p4 -vvvv ../src/testsuite/buildok/seven.stp The previous command's output included: pattern 'detach_pid' matches function 'detach_pid' selected function detach_pid finding prologue for 'detach_pid' entrypc=0xffffffff80143458 highpc=0xffffffff80143499 finding location for local 'task' near address ffffffff8014345f, module bias 0 finding location for local 'type' near address ffffffff8014345f, module bias 0 finding location for local 'task' near address ffffffff8014345f, module bias 0 finding location for local 'type' near address ffffffff8014345f, module bias 0 Thus, the 'task' local is found. However, the SystemTap translator appears to be unable to handle it. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Suggestions for localizing problems with identifiers in SystemTap. 2006-02-27 2:24 ` Suggestions for localizing problems with identifiers in SystemTap William Cohen @ 2006-02-27 4:40 ` Roland McGrath 2006-02-27 15:57 ` William Cohen 0 siblings, 1 reply; 7+ messages in thread From: Roland McGrath @ 2006-02-27 4:40 UTC (permalink / raw) To: systemtap The place I would start is with the "loc2c-test" program. This is a simple test program built during the systemtap build, but not installed. It uses exactly the same infrastructure for finding and interpreting the debug information to identify variables at requested source locations. loc2c-test takes some standard options (see --help), which are the same as you can use with e.g. eu-addr2line. Use the -k option to look at the current kernel and modules, the same way the systemtap translator does. (The other options make it easy to look at info for different kernels.) The required argument is an address constant, giving the PC location of the probe. The -v output from stap should tell you this. ./loc2c-test -k 0xc012c9b1 This yields: kernel/pid.c (0x11) current_stack_pointer [2e9735] long unsigned int pid_hash [2e9752] hlist_head*[] pidhash_shift [2e9763] int pidmap_array [2e9784] pidmap_t[] pidmap_lock [2e9795] spinlock_t __kcrctab_find_task_by_pid_type[2e97a6] long unsigned int const __kstrtab_find_task_by_pid_type[2e97c7] char[] const __ksymtab_find_task_by_pid_type[2e97dd] kernel_symbol const console_printk [2e97f9] int[] mmu_cr4_features [2e9806] long unsigned int xtime [2e9813] timespec time_status [2e9820] int time_maxerror [2e982d] long int time_esterror [2e983a] long int time_adjust [2e9847] long int contig_page_data [2e9854] pglist_data malloc_sizes [2e986d] cache_sizes[] last_pid [2e987a] int init_task [2e988c] task_struct irq_desc [2e98aa] irq_desc_t[] irq_vector [2e98c7] u8[] io_apic_irqs [2e98d4] long unsigned int per_cpu__rcu_data [2e98e1] rcu_data per_cpu__rcu_bh_data [2e98ee] rcu_data rcu_ctrlblk [2e98fb] rcu_ctrlblk rcu_bh_ctrlblk [2e9908] rcu_ctrlblk dcache_lock [2e9915] spinlock_t acpi_noirq [2e9922] int acpi_disabled [2e992f] int acpi_ht [2e993c] int acpi_pci_disabled [2e9949] int skip_ioapic_setup [2e9956] int zone_table [2e996e] zone*[] mem_map [2e997c] page* swapper_space [2e998a] address_space per_cpu__cpu_gdt_table [2e99a8] desc_struct[] default_ldt [2e99c0] desc_struct[] dma_spin_lock [2e99cd] spinlock_t nr_kernel_pages [2e99da] long unsigned int pid_max [2e99e7] int pid_max_min [2e99f9] int pid_max_max [2e9a0b] int __crc_find_task_by_pid_type [2e9a1d] <no type>* detach_pid (0x2e): 0xc012c9b1 (kernel/pid.c:193) .. 0xc012c9ed (kernel/pid.c:208) task [2e9436] task_t* type [2e9445] pid_type tmp [2e9454] int nr [2e9463] int The shows the DWARF scopes that contain the PC address for the probe point, and the variable names defined in each. Here you see the file (global) scope, and inside it the "detach_pid" function scope. If a variable is not shown here, then it wasn't found in the DWARF info. You can use readelf or eu-readelf to investigate something you find missing here. Thanks, Roland ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Suggestions for localizing problems with identifiers in SystemTap. 2006-02-27 4:40 ` Roland McGrath @ 2006-02-27 15:57 ` William Cohen 0 siblings, 0 replies; 7+ messages in thread From: William Cohen @ 2006-02-27 15:57 UTC (permalink / raw) To: Roland McGrath; +Cc: systemtap Roland McGrath wrote: > The place I would start is with the "loc2c-test" program. This is a simple > test program built during the systemtap build, but not installed. It uses > exactly the same infrastructure for finding and interpreting the debug > information to identify variables at requested source locations. > > loc2c-test takes some standard options (see --help), which are the same as > you can use with e.g. eu-addr2line. Use the -k option to look at the > current kernel and modules, the same way the systemtap translator does. > (The other options make it easy to look at info for different kernels.) > The required argument is an address constant, giving the PC location of the > probe. The -v output from stap should tell you this. > > ./loc2c-test -k 0xc012c9b1 Hi Roland, Thanks. for the explanation on how to use loc2c-test. It is pretty easy to get the address of the "function of interest" in the kernel with something like the following: nm /usr/lib/debug/lib/modules/`uname -r`/vmlinux \ |grep " detach_pid$" |awk '{print "0x" $1}' How would one get the addresses for inlined functions and functions in modules? -Will ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2006-02-27 15:57 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2006-02-20 16:40 Summary of nightly tests 20060220 William Cohen 2006-02-20 19:52 ` Martin Hunt 2006-02-20 20:05 ` William Cohen 2006-02-20 20:36 ` Frank Ch. Eigler 2006-02-27 2:24 ` Suggestions for localizing problems with identifiers in SystemTap William Cohen 2006-02-27 4:40 ` Roland McGrath 2006-02-27 15:57 ` William Cohen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).