Summary of nightly tests 20060220

Summary of nightly tests 20060220

[William Cohen]

The most severe problem is the FC i686 2.6.15-1.1831_FC4 kernel double
faults on one of the test systemtap.samples/poll_map.exp

One minor annoyance of the stap translator is that it prints
out the same error messages multiple times on the tests that fail.

FC4 i686 Linux 2.6.15-1.1831_FC4

	(double fault in systemtap.samples/poll_map.exp)



FC5 i686 Linux 2.6.15-1.1948_FC5smp

Stap translator summary of failed tests
buildok-seven.stp:
	Unable to find identifier $type in function find_pid
	Unable to find identifier $task in function detach_pid
buildok-syscall.stp:
	Unable to find identifier $timeout in function sys_poll (syscall2.stp)
	Unable to find identifier $regs in function sys_rt_sigsuspend 
(syscall2.stp)

Testsuite summary of failed tests
		=== systemtap Summary ===

# of expected passes		129



FC5 x86_64 Linux 2.6.15-1.1948_FC5 (1 processor)

Stap translator summary of failed tests
buildok-seven.stp: unable to find identifier $task for function detach_pid
buildok-syscall.stp: unable to resolve function sys_mmap2

Testsuite summary of failed tests
		=== systemtap Summary ===

# of expected passes		128
# of unsupported tests		1



RHEL4U3 i686 Linux 2.6.9-30.EL

Stap translator summary of failed tests
buildok-syscall.stp:
	cannot resolve $regs used in tapset/syscalls2.stp (sys_rt_sigsuspend)


Testsuite summary of failed tests
		=== systemtap Summary ===

# of expected passes		128
# of unsupported tests		1



RHEL4U3 x86_64 Linux 2.6.9-30.ELsmp

Stap translator summary of failed tests
buildok-syscall.stp: unable to resolve function sys_mmap2

Testsuite summary of failed tests
FAIL: ./systemtap.stress/current.stp startup (eof): test harness problem
		=== systemtap Summary ===

# of expected passes		126
# of unexpected failures	1

Re: Summary of nightly tests 20060220

[Martin Hunt]

> FC5 i686 Linux 2.6.15-1.1948_FC5smp
> 
> Stap translator summary of failed tests
> buildok-seven.stp:
> 	Unable to find identifier $type in function find_pid
> 	Unable to find identifier $task in function detach_pid

There don't appear to be significant changes in the sources to account
for this. That makes it a likely gcc bug. 

Can someone write up the proper procedure to use to report these? I
think it should be on the web page.

Martin

Re: Summary of nightly tests 20060220

[William Cohen]

Martin Hunt wrote:
>>FC5 i686 Linux 2.6.15-1.1948_FC5smp
>>
>>Stap translator summary of failed tests
>>buildok-seven.stp:
>>	Unable to find identifier $type in function find_pid
>>	Unable to find identifier $task in function detach_pid
> 
> 
> There don't appear to be significant changes in the sources to account
> for this. That makes it a likely gcc bug. 
> 
> Can someone write up the proper procedure to use to report these? I
> think it should be on the web page.
> 
> Martin
> 
> 

The problem appears only in certain kernels. Possible causes for this 
type of problem:

1) source code has changed so those variable/arguments don't exist
2) gcc tool chain not generating the proper debug information
3) stap is not finding them in the debugging information

Are there other possibilities or should this be finer grain resolution, 
e.g. elfutils vs stap translator? The procedure should make it clear 
which of the alternatives can be ruled out. Ideally, would like it to 
point to to which one is causing the problem.

-Will

Re: Summary of nightly tests 20060220

[Frank Ch. Eigler]

hunt wrote:

> > buildok-seven.stp:
> > 	Unable to find identifier $type in function find_pid
> > 	Unable to find identifier $task in function detach_pid
> 
> There don't appear to be significant changes in the sources to
> account for this. That makes it a likely gcc bug. [...]

Yes, possibly.  We will likely have a RH compiler person working on an
ongoing basis to fix such problems.

I can't find a message I remember writing to the mailing list about
the change I committed to buildok/seven.stp on 2006-02-07.  So to
recap: It adds the "-u" stap flag to force all $target variables to be
expanded.  This, while valuable to some extent in exposing debuginfo
problems, was meant to be a stopgap measure.  To be considered
complete, the system calls tapset should eventually come with a test
case that exercises all the auxiliary / embedded-C functions, at least
at the buildok level.  (This is what src/HACKING suggests.)  At that
point, I would remove "-u", and let unresolved $target variables pass
quietly.

- FChE

Suggestions for localizing problems with identifiers in SystemTap.

[William Cohen]

Martin Hunt wrote:
>>FC5 i686 Linux 2.6.15-1.1948_FC5smp
>>
>>Stap translator summary of failed tests
>>buildok-seven.stp:
>>	Unable to find identifier $type in function find_pid
>>	Unable to find identifier $task in function detach_pid
> 
> 
> There don't appear to be significant changes in the sources to account
> for this. That makes it a likely gcc bug. 
> 
> Can someone write up the proper procedure to use to report these? I
> think it should be on the web page.
> 
> Martin

I have jotted down some notes on how to track down these types of 
problems with the identifiers. The notes are below. There are still lots 
of holes.

-Will



Suggestions for localizing problems with identifiers in SystemTap.


Possible causes for this problems symbols in SystemTap scripts:

1) source code doesn't have function, variable, or argument
2) GCC tool chain not generating the proper debug information
3) system tap translator is unable them in the debugging information

-----

Source code doesn't have function, variable, or argument

This can be determined by examining the source code.

1) Find out which kernel is being used and location of source
code. You will need to have the cscope rpm installed

	uname -r

	get the matching source SRPM.
	install srpm on  the machine with:

	rpm -Uvh kernel-<version>.src.rpm

	cd into SPEC directory
	rpmbuild -bp kernel-2.6.rpm

	cd into BUILD/kernel-<version>/linux-<version> directory
	make mrproper
	cp /boot/config-`uname -r` .config; make oldconfig; make cscope

2) Find where the problem function is located and examine code for
variable or arument name. Ccsope is extremely useful for navigating
locating the functions.

	Start up cscope and use the following cscope entries:
		"Find this global definition:" or "Find this egrep pattern:"
	Type in function name or pattern
	If function shows up, hit return to look at function by hitting enter.
		cscope will start an editor. Take a look for the problem symbol
		if the function is "static" or "static inline",
			the compiler could optimize these away if not called.
			cscope can also show where the function is used

-----

GCC toolchain not generating the proper debug information

After establishing that the the symbol exists in source code,
determine whether the compiler is generating the proper debug
information.  The program "readelf" in the binutils RPM can be used
the examine the debugging information in the kernel and modules.

readelf --debug-dump /usr/lib/debug/lib/modules/`uname -r`/vmlinux > 
/tmp/debuginfo

This will generate a very large file /tmp/debuginfo that can be
examined.  For example want to find out why $task is not being found
in function detach_pid for src/testsuite/buildok/seven.stp.

Looking for the section of the output in /tmp/debuginfo for the
detach_pid function we find the following section:


  <1><3554ae>: Abbrev Number: 83 (DW_TAG_subprogram)
      DW_AT_sibling     : <35550b>
      DW_AT_external    : 1
      DW_AT_name        : (indirect string, offset: 0x1c783): detach_pid
      DW_AT_decl_file   : 1
      DW_AT_decl_line   : 193
      DW_AT_prototyped  : 1
      DW_AT_low_pc      : 0xffffffff80143458
      DW_AT_high_pc     : 0xffffffff80143499
      DW_AT_frame_base  : 0x8ee43        (location list)
  <2><3554cf>: Abbrev Number: 76 (DW_TAG_formal_parameter)
      DW_AT_name        : (indirect string, offset: 0xba9d3): task
      DW_AT_decl_file   : 1
      DW_AT_decl_line   : 192
      DW_AT_type        : <35184d>
      DW_AT_location    : 0x8eea3        (location list)
<2><3554de>: Abbrev Number: 76 (DW_TAG_formal_parameter)
      DW_AT_name        : (indirect string, offset: 0x4fb9b): type
      DW_AT_decl_file   : 1
      DW_AT_decl_line   : 192
      DW_AT_type        : <35062a>
      DW_AT_location    : 0x8eec6        (location list)
  <2><3554ed>: Abbrev Number: 72 (DW_TAG_variable)
      DW_AT_name        : tmp
      DW_AT_decl_file   : 1
      DW_AT_decl_line   : 194
      DW_AT_type        : <34e505>
      DW_AT_location    : 0x8ef0f        (location list)
  <2><3554fc>: Abbrev Number: 72 (DW_TAG_variable)
      DW_AT_name        : nr
      DW_AT_decl_file   : 1
      DW_AT_decl_line   : 194
      DW_AT_type        : <34e505>
      DW_AT_location    : 0x8ef45        (location list)


It is clear the "task" identifier is in the debugging information and
that it is a argument being passed into the function.  The type of the
identifier can be determined by by searching for the "DW_AT_type", in
this case "><35184d>":


<1><35184d>: Abbrev Number: 7 (DW_TAG_pointer_type)
      DW_AT_byte_size   : 8
      DW_AT_type        : <350f16>

The chain can be followed searching for "><350f16>":

  <1><350f16>: Abbrev Number: 6 (DW_TAG_typedef)
      DW_AT_name        : (indirect string, offset: 0x782ad): task_t
      DW_AT_decl_file   : 9
      DW_AT_decl_line   : 184
      DW_AT_type        : <34e8a0>

Searching for "><34e8a0>" yield task "task_struct" followed by its fields:


  <1><34e8a0>: Abbrev Number: 15 (DW_TAG_structure_type)
      DW_AT_sibling     : <34f062>
      DW_AT_name        : (indirect string, offset: 0x15daa): task_struct
      DW_AT_byte_size   : 1888
      DW_AT_decl_file   : 10
      DW_AT_decl_line   : 13
  <2><34e8ad>: Abbrev Number: 16 (DW_TAG_member)
      DW_AT_name        : (indirect string, offset: 0xb09cc): state
      DW_AT_decl_file   : 9
      DW_AT_decl_line   : 701
      DW_AT_type        : <34f60a>
      DW_AT_data_member_location: 2 byte block: 23 0 
(DW_OP_plus_uconst: 0)
  <2><34e8bc>: Abbrev Number: 16 (DW_TAG_member)
      DW_AT_name        : (indirect string, offset: 0x8efe): thread_info
      DW_AT_decl_file   : 9
      DW_AT_decl_line   : 702
      DW_AT_type        : <35191d>
      DW_AT_data_member_location: 2 byte block: 23 8 
(DW_OP_plus_uconst: 8)


-----

SystemTap (stap) is not finding the identiers in the debugging information

After eliminating the source code and compiler, the SystemTap
translator is the likely suspect. The systemtap translator might be
able to find the identifier but is unable to process it. For example
src/testsuite/buildok/seve.stp instruments the detach_pid funtion in
the kernel and on the x86_64 produces the following error message:

semantic error: unresolved target-symbol expression: identifier '$task' 
at 
/home/wcohen/stap_testing_200602261550/src/testsuite/buildok/seven.stp:19:35


One can use one (or more) "-v" (verbose) options with the SystemTap
translator to get get a clearer picture of what the translator is
doing and getting some idea of why the symbol is not being resolved
correctly. The following command was used:

stap -p4 -vvvv ../src/testsuite/buildok/seven.stp

The previous command's output included:

pattern 'detach_pid' matches function 'detach_pid'
selected function detach_pid
finding prologue for 'detach_pid' entrypc=0xffffffff80143458 
highpc=0xffffffff80143499
finding location for local 'task' near address ffffffff8014345f, module 
bias 0
finding location for local 'type' near address ffffffff8014345f, module 
bias 0
finding location for local 'task' near address ffffffff8014345f, module 
bias 0
finding location for local 'type' near address ffffffff8014345f, module 
bias 0

Thus, the 'task' local is found. However, the SystemTap translator
appears to be unable to handle it.

Re: Suggestions for localizing problems with identifiers in SystemTap.

[Roland McGrath]

The place I would start is with the "loc2c-test" program.  This is a simple
test program built during the systemtap build, but not installed.  It uses
exactly the same infrastructure for finding and interpreting the debug
information to identify variables at requested source locations.

loc2c-test takes some standard options (see --help), which are the same as
you can use with e.g. eu-addr2line.  Use the -k option to look at the
current kernel and modules, the same way the systemtap translator does.
(The other options make it easy to look at info for different kernels.)
The required argument is an address constant, giving the PC location of the
probe.  The -v output from stap should tell you this.

	./loc2c-test -k 0xc012c9b1

This yields:

    kernel/pid.c (0x11)
        current_stack_pointer         [2e9735]	long unsigned int
        pid_hash                      [2e9752]	hlist_head*[]
        pidhash_shift                 [2e9763]	int
        pidmap_array                  [2e9784]	pidmap_t[]
        pidmap_lock                   [2e9795]	spinlock_t
        __kcrctab_find_task_by_pid_type[2e97a6]	long unsigned int const
        __kstrtab_find_task_by_pid_type[2e97c7]	char[] const
        __ksymtab_find_task_by_pid_type[2e97dd]	kernel_symbol const
        console_printk                [2e97f9]	int[]
        mmu_cr4_features              [2e9806]	long unsigned int
        xtime                         [2e9813]	timespec
        time_status                   [2e9820]	int
        time_maxerror                 [2e982d]	long int
        time_esterror                 [2e983a]	long int
        time_adjust                   [2e9847]	long int
        contig_page_data              [2e9854]	pglist_data
        malloc_sizes                  [2e986d]	cache_sizes[]
        last_pid                      [2e987a]	int
        init_task                     [2e988c]	task_struct
        irq_desc                      [2e98aa]	irq_desc_t[]
        irq_vector                    [2e98c7]	u8[]
        io_apic_irqs                  [2e98d4]	long unsigned int
        per_cpu__rcu_data             [2e98e1]	rcu_data
        per_cpu__rcu_bh_data          [2e98ee]	rcu_data
        rcu_ctrlblk                   [2e98fb]	rcu_ctrlblk
        rcu_bh_ctrlblk                [2e9908]	rcu_ctrlblk
        dcache_lock                   [2e9915]	spinlock_t
        acpi_noirq                    [2e9922]	int
        acpi_disabled                 [2e992f]	int
        acpi_ht                       [2e993c]	int
        acpi_pci_disabled             [2e9949]	int
        skip_ioapic_setup             [2e9956]	int
        zone_table                    [2e996e]	zone*[]
        mem_map                       [2e997c]	page*
        swapper_space                 [2e998a]	address_space
        per_cpu__cpu_gdt_table        [2e99a8]	desc_struct[]
        default_ldt                   [2e99c0]	desc_struct[]
        dma_spin_lock                 [2e99cd]	spinlock_t
        nr_kernel_pages               [2e99da]	long unsigned int
        pid_max                       [2e99e7]	int
        pid_max_min                   [2e99f9]	int
        pid_max_max                   [2e9a0b]	int
        __crc_find_task_by_pid_type   [2e9a1d]	<no type>*
        detach_pid (0x2e): 0xc012c9b1 (kernel/pid.c:193) .. 0xc012c9ed (kernel/pid.c:208)
            task                          [2e9436]	task_t*
            type                          [2e9445]	pid_type
            tmp                           [2e9454]	int
            nr                            [2e9463]	int

The shows the DWARF scopes that contain the PC address for the probe point,
and the variable names defined in each.  Here you see the file (global)
scope, and inside it the "detach_pid" function scope.  If a variable is not
shown here, then it wasn't found in the DWARF info.  You can use readelf or
eu-readelf to investigate something you find missing here.  


Thanks,
Roland

Re: Suggestions for localizing problems with identifiers in SystemTap.

[William Cohen]

Roland McGrath wrote:
> The place I would start is with the "loc2c-test" program.  This is a simple
> test program built during the systemtap build, but not installed.  It uses
> exactly the same infrastructure for finding and interpreting the debug
> information to identify variables at requested source locations.
> 
> loc2c-test takes some standard options (see --help), which are the same as
> you can use with e.g. eu-addr2line.  Use the -k option to look at the
> current kernel and modules, the same way the systemtap translator does.
> (The other options make it easy to look at info for different kernels.)
> The required argument is an address constant, giving the PC location of the
> probe.  The -v output from stap should tell you this.
> 
> 	./loc2c-test -k 0xc012c9b1

Hi Roland,

Thanks. for the explanation on how to use loc2c-test. It is pretty easy 
to get the address of the "function of interest" in the kernel with 
something like the following:

  nm /usr/lib/debug/lib/modules/`uname -r`/vmlinux \
  |grep " detach_pid$" |awk '{print "0x" $1}'

How would one get the addresses for inlined functions and functions in 
modules?

-Will