Follow-up to [Bug translator/14596] New: probe module enhancement request

Follow-up to [Bug translator/14596] New: probe module enhancement request

[Chaiken, Alison]

Colleagues, my co-worker Jim Baxter and I have been trying to get cross-compiled SystemTap scripts to work with modules on Freescale ARMv7 i.MX6.    First I compiled and ran several examples on the target board from http://sourceware.org/systemtap/examples/  just to make sure that the whole machinery was working. The compilation host is an Ubuntu VM.

My goal involves probing modules, so next I tried compiling the file probedrm.stp

      probe module("drm").function("*")
      {print("I am here\n"); exit();}

using the following script

---

STAP_SYSROOT="/build/meibp-2013/build/tmp/sysroots"
CROSS_COMPILE=arm-none-linux-gnueabi-
STP_FILE=probedrm.stp
STP_BASE_NAME=$(basename ${STP_FILE} .stp)

${STAP_SYSROOT}/x86_64-linux/usr/bin/stap $1 -a arm -v -g \
                -R ${STAP_SYSROOT}/mx6q/usr/share/systemtap/runtime \
                --sysroot=${STAP_SYSROOT}/mx6q \
                -B CROSS_COMPILE=${CROSS_COMPILE} \
                -r ${STAP_SYSROOT}/mx6q/usr/src/kernel \
                ${STP_FILE} -d ${STP_BASE_NAME} -m ${STP_BASE_NAME} -p4

----

The result is 

---

[achaiken@sb-ubuntu-1204-64bit systemtap]$ ./compilemodule.sh
Pass 1: parsed user script and 83 library script(s) using 55172virt/22172res/2112shr/20696data kb, in 90usr/10sys/104real ms.
semantic error: while resolving probe point: identifier 'module' at probedrm.stp:1:7
        source: probe module("drm").function("*")
                      ^
semantic error: no match
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 120112virt/23456res/2436shr/21592data kb, in 0usr/80sys/86real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.

----

I've tried various configurations of the "-d" and "-r" options as suggested at Bug 14596.     Is the problem that the compiler persistently looks in the localhost /lib/modules?    Given that the target is fairly beefy in computational power, is running stap compiler natively likely an easier strategy in the long run?    

Thanks for any suggestions,
Alison Chaiken
Mentor Embedded Software Division

Re: Follow-up to [Bug translator/14596] New: probe module enhancement request

[Frank Ch. Eigler]

"Chaiken, Alison" <Alison_Chaiken@mentor.com> writes:

> [...]
>       probe module("drm").function("*")
>       {print("I am here\n"); exit();}
> using the following script
> [...]
> ${STAP_SYSROOT}/x86_64-linux/usr/bin/stap $1 -a arm [...]
> [...]
> semantic error: while resolving probe point: identifier 'module' at probedrm.stp:1:7
>         source: probe module("drm").function("*")
>                       ^
> [...]
> Is the problem that the compiler persistently looks in the localhost /lib/modules?

It shouldn't do that; an strace should clear up where it's looking, and a 
stap "--vp 04000" may be enough to give extra information.  Maybe something
as simple as the drm.ko file not carrying CONFIG_DEBUG_INFO=y.

> Given that the target is fairly beefy in computational power, is
> running stap compiler natively likely an easier strategy in the long
> run?

They should both be possible.

- FChE

Follow-up to [Bug translator/14596] New: probe module enhancement request

[Chaiken, Alison]

Alison:
Thanks for the quick response.

FChE:
> It shouldn't do that; an strace should clear up where it's looking, and a
> stap "--vp 04000" may be enough to give extra information.


Here's the invocation:

${STAP_SYSROOT}/x86_64-linux/usr/bin/stap --vp 04000 $1 -a arm -m ${STP_BASE_NAME} -p4 \
-R ${STAP_SYSROOT}/mx6q/usr/share/systemtap/runtime \
--sysroot=${STAP_SYSROOT}/mx6q \
-B CROSS_COMPILE=${CROSS_COMPILE} \
-r ${STAP_SYSROOT}/mx6q/usr/src/kernel \
${STP_FILE}

where mx6q is our target's arch.

Here's the output:


Processing probedrm.stp
blacklist regexps:
blfn: ^(atomic_notifier_call_chain|default_do_nmi|__die|die_nmi|do_debug|do_general_protection|do_int3|do_IRQ|do_page_fault|do_sparc64_fault|do_trap|dummy_nmi_callback|flush_icache_range|ia64_bad_break|ia64_do_page_fault|ia64_fault|io_check_error|mem_parity_error|nmi_watchdog_tick|notifier_call_chain|oops_begin|oops_end|program_check_exception|single_step_exception|sync_regs|unhandled_fault|unknown_nmi_error|xen_[gs]et_debugreg|xen_irq_.*|xen_.*_fl_direct.*|check_events|xen_adjust_exception_frame|xen_iret.*|xen_sysret64.*|test_ti_thread_flag.*|inat_get_opcode_attribute|system_call_after_swapgs|.*raw_.*_lock.*|.*raw_.*_unlock.*|.*raw_.*_trylock.*|.*read_lock.*|.*read_unlock.*|.*read_trylock.*|.*write_lock.*|.*write_unlock.*|.*write_trylock.*|.*write_seqlock.*|.*write_sequnlock.*|.*spin_lock.*|.*spin_unlock.*|.*spin_trylock.*|.*spin_is_locked.*|rwsem_.*lock.*|.*mutex_.*lock.*|raw_.*|atomic_.*|atomic64_.*|get_bh|put_bh|.*apic.*|.*APIC.*|.*softirq.*|.*IRQ.*|.*_intr.*|__delay|.*kernel_text.*|get_current|current_.*|.*exception_tables.*|.*setup_rt_frame.*|.*preempt_count.*|preempt_schedule|special_mapping_.*|.*_pte_.*)$
blfn_ret: ^(do_exit|sys_exit|sys_exit_group)$
blfile: ^(kernel/kprobes\.c|arch/.*/kernel/kprobes\.c|.*/include/asm/io\.h|.*/include/asm/io_64\.h|.*/include/asm/bitops\.h|drivers/ide/ide-iops\.c|arch/.*/kernel/paravirt\.c|.*/include/asm/paravirt\.h|fs/seq_file\.c)$
blsection: ^(\.init\.|\.exit\.|\.devinit\.|\.devexit\.|\.cpuinit\.|\.cpuexit\.|\.meminit\.|\.memexit\.)
dwarf_builder::build for drm
parse '*', func '*'
semantic error: while resolving probe point: identifier 'module' at probedrm.stp:1:7
source: probe module("drm").function("*")


Hopefully that's more suggestive of a root cause to one of you than to me!   (By the way, I also tried '--vp 04000' and "--vp 04000" and tried putting "--vp 04000" at different places in the script, but
 I always got "unrecognized option" as a response.)

FChE:
> Maybe something as simple as the drm.ko file not carrying CONFIG_DEBUG_INFO=y.

I checked and  it seems to be set:

grep DEBUG tmp/work/mx6q-mel-linux-gnueabi/linux-imx-3.5.7.13+gitr1+0e9463fac8d9eb9812fe571edd97ebce88055755-r15.1/defconfig

# CONFIG_SLUB_DEBUG is not set
CONFIG_PM_DEBUG=y
# CONFIG_SCHED_DEBUG is not set
# CONFIG_DEBUG_BUGVERBOSE is not set
CONFIG_DEBUG_INFO=y


Thanks for any further suggestions,
Alison

Re: Follow-up to [Bug translator/14596] New: probe module enhancement request

[Frank Ch. Eigler]

Hi -

> [...]
> Here's the output:
> 
> Processing probedrm.stp
> [...]
> parse '*', func '*'
> semantic error: while resolving probe point: identifier 'module' at probedrm.stp:1:7
> source: probe module("drm").function("*") [...]

Hm, so a drm.ko file was not even found apparently. 

What does % strace stap .... 2>&1 | grep -i drm.ko    say?
What about   % stap .... -L 'module("*").function("*")' ?


- FChE

RE: Follow-up to [Bug translator/14596] New: probe module enhancement request

[Chaiken, Alison]

FChE responds:
Hm, so a drm.ko file was not even found apparently.
What does % strace stap .... 2>&1 | grep -i drm.ko    say?

Alison:
Nothing.    I dumped the strace output into a file and searched over it, and there are no occurrences of "drm.ko".    Note that drm.ko is not a particular problem; that module was picked at random.    There is no module that works.

FChE continues:
What about   % stap .... -L 'module("*").function("*")' ?

I can't figure out how to invoke a command with nested quotes, either from CLI or from a script:

   stap: invalid option -- ' ' 

results with -L 'module("*").function("*")'.

Reversing the single and double quotes and trying again produces

   syntax error near unexpected token `('

as did no quotes at all.   Perhaps this is due to the stap version?

   [achaiken@sb-ubuntu-1204-64bit systemtap]$ stap --version
   Systemtap translator/driver (version 1.6/0.152 non-git sources)

Presumably the most likely problem of the overall compilation failure is that stap doesn't know the path to search for the compiled kernel modules, and that I need to configure it to look in the right place, perhaps with the "-d" or "--sysenv" switch?

We're using Yocto to cross-compile, so perhaps not a lot of others have tried that particular use case yet?

Thanks again,
Alison Chaiken
Mentor Embedded Software Division

Re: Follow-up to [Bug translator/14596] New: probe module enhancement request

[Frank Ch. Eigler]

Hi -

> > What does % strace stap .... 2>&1 | grep -i drm.ko    say?

> Nothing.  I dumped the strace output into a file and searched over
> it, and there are no occurrences of "drm.ko".  Note that drm.ko is
> not a particular problem; that module was picked at random.  There
> is no module that works.

OK.  Does the strace give a hint at where stap *is* looking for the
modules?  There should be a bunch of stat / openat / getdents type
syscalls.


> FChE continues:
> What about   % stap .... -L 'module("*").function("*")' ?

> I can't figure out how to invoke a command with nested quotes,
> either from CLI or from a script:

You could just run it directly as I typed it.

% stap <<<YOUR OTHER -r/-a/-B/etc. OPTIONS HERE>>> -L 'module("*").function("*")'

>    [achaiken@sb-ubuntu-1204-64bit systemtap]$ stap --version
>    Systemtap translator/driver (version 1.6/0.152 non-git sources)

(You may be able to build or get hold of a newer version; 
that's two years old.)


> Presumably the most likely problem of the overall compilation
> failure is that stap doesn't know the path to search for the
> compiled kernel modules [...]

You're right.  The question is how to trick that old version into
doing what you need.


- FChE

RE: Follow-up to [Bug translator/14596] New: probe module enhancement request

[Chaiken, Alison]

[-- Attachment #1: Type: text/plain, Size: 2035 bytes --]

FChE:
Does the strace give a hint at where stap *is* looking for the
modules?  There should be a bunch of stat / openat / getdents type
syscalls.

Alison:
A gzipped version of the full output is attached.   In general, stap does seem to be looking in the correct sysroot, 
/build/meibp-2013/build/tmp/sysroots/mx6q.     It's not obvious to me where it's going wrong.   You'd be inclined to suspect the very last directory the script looks in

   [achaiken@sb-ubuntu-1204-64bit systemtap]$ ls /build/meibp-2013/build/tmp/sysroots/mx6q/usr/src/kernel/init
   do_mounts.h  Kconfig  Makefile

but that looks okay.   There is no reference to /lib/modules before the failure, hence my puzzlement.

FChE continues:
% stap <<<YOUR OTHER -r/-a/-B/etc. OPTIONS HERE>>> -L 'module("*").function("*")'

Alison:
This

    #!/bin/sh
    STAP_SYSROOT="/build/meibp-2013/build/tmp/sysroots"
    CROSS_COMPILE=arm-none-linux-gnueabi-

    ${STAP_SYSROOT}/x86_64-linux/usr/bin/stap -a arm \
		-R ${STAP_SYSROOT}/mx6q/usr/share/systemtap/runtime \
		--sysroot=${STAP_SYSROOT}/mx6q \
		-B CROSS_COMPILE=${CROSS_COMPILE} \
		-r ${STAP_SYSROOT}/mx6q/usr/src/kernel \
		-L 'module("*").function("*")' 

produces *no* output.  Hopefully that invocation is now what you intended!  I see that "man stap" says that -r should point to where /lib/modules/RELEASE/build is, but 
/build/meibp-2013/build/tmp/sysroots/mx6q/lib/modules/3.5.7.13-01716-g0e9463fa 
has no build directory:

    [achaiken@sb-ubuntu-1204-64bit systemtap]$ ls /build/meibp-2013/build/tmp/sysroots/mx6q/lib/modules   /3.5.7.13-01716-g0e9463fa/
    kernel/  modules.builtin  modules.order

Perhaps that then is the crux of the matter?

Version 1.6/0.152 is what is apparently packaged for Ubuntu 12.04 (which is our default VM runtime).    I can try and compile a newer version if you think that will solve the problem.

Thanks again,
Alison 
(who apologizes for using employer-mandated Outlook, but only as a web app running on $DEITY's Debian)


[-- Attachment #2: stap_drm_strace.out.gz --]
[-- Type: application/binary, Size: 276361 bytes --]

RE: Follow-up to [Bug translator/14596] New: probe module enhancement request

[Chaiken, Alison]

At https://github.com/fche/systemtap/releases, is there any place to download checksum files?   Our build system enjoys them.

Thanks,
Alison

Re: Follow-up to [Bug translator/14596] New: probe module enhancement request

[Mark Wielaard]

On Wed, Jul 03, 2013 at 07:15:19AM +0000, Chaiken, Alison wrote:
> At https://github.com/fche/systemtap/releases, is there any place to
> download checksum files?   Our build system enjoys them.

There is ftp://sourceware.org/pub/systemtap/releases/ which contain
newer releases. There is an md5 sum file there. If you are using git
then you can verify the release tags with something like:
$ git tag --verify release-2.2.1

Cheers,

Mark