staplog: Failed to find _stp_relay_data in module 'counter'

staplog: Failed to find _stp_relay_data in module 'counter'

[Timo Juhani Lindfors]

Hi,

I was planning to enable the systemtap staplog crash extension in the
debian package but I can't figure out how to get it to work. Here's what
I'm currently trying:

1) start stap with

stap -m counter -e 'global i; probe timer.ms(1000) { printf("i = %d\n", i++); }'

2) take dump of the system memory using "xm dump-core lindi3 core"

3) open the dump with "crash -e emacs
   /usr/lib/debug/boot/vmlinux-3.8-2-amd64 core" and try to use staplog:

crash 7.0.1
Copyright (C) 2002-2013  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

      KERNEL: /usr/lib/debug/boot/vmlinux-3.8-2-amd64
    DUMPFILE: core
        CPUS: 4
        DATE: Mon Jul 15 12:00:24 2013
      UPTIME: 54 days, 12:58:55
LOAD AVERAGE: 0.00, 0.10, 0.37
       TASKS: 185
    NODENAME: lindi3
     RELEASE: 3.8-2-amd64
     VERSION: #1 SMP Debian 3.8.13-1
     MACHINE: x86_64  (3210 Mhz)
      MEMORY: 1 GB
       PANIC: ""
         PID: 0
     COMMAND: "swapper/0"
        TASK: ffffffff81614400  (1 of 4)  [THREAD_INFO: ffffffff81600000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)
     WARNING: panic task not found

crash> extend staplog.so
./staplog.so: shared object loaded
crash> staplog counter
staplog: Failed to find _stp_relay_data in module 'counter'.
crash> p &_stp_relay_data
No symbol "_stp_relay_data" in current context.
p: gdb request failed: p &_stp_relay_data
crash> mod -s counter counter.ko
     MODULE       NAME              SIZE  OBJECT FILE
ffffffffa024b330  counter          31959  counter.ko 
crash> p &_stp_relay_data
$1 = (<data variable, no debug info> *) 0xffffffffa024b580 <_stp_relay_data>
crash> staplog counter
staplog: Failed to find _stp_relay_data in module 'counter'.
crash> 


Any idea what's wrong?


With gdb I see that

symbol_value_module("_stp_relay_data", module);

fails. I set a breakpoint at the "for ( ; sp < sp_end; sp++) {" line in
crash and noticed that even though it finds a lot of symbols
"_stp_relay_data" is not one of them:

1: sp->name = 0x22318d0 "_MODULE_START_counter"
1: sp->name = 0x2231307 "paravirt_read_tsc"
1: sp->name = 0x2231328 "atomic_inc"
1: sp->name = 0x2231333 "atomic_dec"
1: sp->name = 0x223133e "atomic_add_return"
1: sp->name = 0x2231350 "__stp_relay_dropped_open"
1: sp->name = 0x2231369 "skip_atoi"
1: sp->name = 0x2231373 "number"
1: sp->name = 0x223137a "number_size"
1: sp->name = 0x2231386 "_stp_vsprint_char_size"
1: sp->name = 0x223139d "_stp_vsprint_char"
1: sp->name = 0x22313af "check_binary_precision"
1: sp->name = 0x22313c6 "_stp_vsprint_binary"
1: sp->name = 0x22313da "param_get_int64_t"
1: sp->name = 0x22313ec "param_set_int64_t"
1: sp->name = 0x22313fe "copy_from_user_page_noflush"
1: sp->name = 0x223141a "copy_to_user_page_noflush"
1: sp->name = 0x2231434 "cpumask_next"
1: sp->name = 0x2231441 "_stp_module_panic_notifier"
1: sp->name = 0x223145c "_stp_kmodule_update_address"
1: sp->name = 0x2231478 "_stp_vsprint_memory_size"
1: sp->name = 0x2231491 "_stp_runtime_contexts_free"
1: sp->name = 0x22314ac "_stp_mempool_destroy"
1: sp->name = 0x22314c1 "_stp_unlock_transport_dir"
1: sp->name = 0x22314db "__stp_relay_remove_buf_file_callback"
1: sp->name = 0x2231500 "_stp_lock_transport_dir"
1: sp->name = 0x2231518 "_stp_remove_root_dir"
1: sp->name = 0x223152d "__stp_relay_create_buf_file_callback"
1: sp->name = 0x2231552 "__stp_relay_dropped_read"
1: sp->name = 0x223156b "hweight_long"
1: sp->name = 0x2231578 "_stp_ctl_open_cmd"
1: sp->name = 0x223158a "copy_from_user"
1: sp->name = 0x2231599 "_stp_vsprint_binary_size"
1: sp->name = 0x22315b2 "_stp_transport_data_fs_stop"
1: sp->name = 0x22315ce "__stp_relay_subbuf_start_callback"
1: sp->name = 0x22315f0 "stp_unlock_probe.constprop.18"
1: sp->name = 0x223160e "_stp_hrtimer_get_interval"
1: sp->name = 0x2231628 "_stp_kmalloc_gfp.constprop.30"
1: sp->name = 0x2231646 "_stp_ctl_close_cmd"
1: sp->name = 0x2231659 "_stp_kmodule_relocate"
1: sp->name = 0x223166f "lookup_bad_addr"
1: sp->name = 0x223167f "_stp_ctl_poll_cmd"
1: sp->name = 0x2231691 "__stp_relay_wakeup_timer"
1: sp->name = 0x22316aa "_stp_mempool_free"
1: sp->name = 0x22316bc "_stp_ctl_read_cmd"
1: sp->name = 0x22316ce "_stp_unregister_ctl_channel"
1: sp->name = 0x22316ea "_stp_vsprint_memory"
1: sp->name = 0x22316fe "_stp_mempool_alloc"
1: sp->name = 0x2231711 "_stp_ctl_send"
1: sp->name = 0x223171f "_stp_ctl_send_notify"
1: sp->name = 0x2231734 "_stp_ctl_work_callback"
1: sp->name = 0x223174b "_stp_alloc_percpu"
1: sp->name = 0x223175d "_stp_transport_data_fs_close"
1: sp->name = 0x223177a "_stp_transport_fs_close"
1: sp->name = 0x2231792 "_stp_transport_data_fs_init"
1: sp->name = 0x22317ae "init_module"
1: sp->name = 0x22317ba "stp_print_flush"
1: sp->name = 0x22317ca "_stp_print_flush"
1: sp->name = 0x22317db "_stp_reserve_bytes"
1: sp->name = 0x22317ee "probe_2049"
1: sp->name = 0x22317f9 "_stp_error"
1: sp->name = 0x2231804 "_stp_build_id_check.constprop.26"
1: sp->name = 0x2231825 "_stp_vsnprintf.constprop.39"
1: sp->name = 0x2231841 "_stp_snprintf.constprop.38"
1: sp->name = 0x223185c "_stp_warn"
1: sp->name = 0x2231866 "_stp_module_notifier"
1: sp->name = 0x223187b "_stp_cleanup_and_exit"
1: sp->name = 0x2231891 "cleanup_module"
1: sp->name = 0x22318a0 "_stp_ctl_write_cmd"
1: sp->name = 0x22318b3 "_stp_hrtimer_notify_function"
1: sp->name = 0x22312f0 "arch_local_irq_restore"
1: sp->name = 0x2231319 "_stp_softerror"


-Timo

Re: staplog: Failed to find _stp_relay_data in module 'counter'

[Frank Ch. Eigler]

Timo Juhani Lindfors <timo.lindfors@iki.fi> writes:

> [...]
>      RELEASE: 3.8-2-amd64
> [...]
> crash> extend staplog.so
> ./staplog.so: shared object loaded
> crash> staplog counter
> staplog: Failed to find _stp_relay_data in module 'counter'.
> [...]

The _stp_relay_data symbol is defined in runtime/transport/relay_v2.c,
which is only one possible configuration on more modern kernels.  The
STP_TRANSPORT_VERSION influences choice of stap transport selection,
as does STP_USE_RING_BUFFER.  Unfortunately staplog.so* has not been
updated with all variants that stap supports.  It is possible that a
different problem is responsible, but this is where I'd start looking.

- FChE