cannot trace user-level code in cross compilation environment

cannot trace user-level code in cross compilation environment

[Zhiwei Ying]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

Hi,

I built a systemtap1.4 on ubuntu 10.4 32 bits. And also cross-compile
staprun to a develop board with kernel 2.6.35 x86.

On the host pc side,

stap -r /target-board/kernel xxx.stp -p 4 -m traceauto.ko

then copy the traceauto.ko to the board,
staprun -u -v -m traceauto.ko -c ./hello


xxx.stp
probe process("/ddtv/tracedrv/test/dhg/hello/hello").function("*").call
{
        printf("USER, %s %s %s\n", thread_indent(1), pp(), $$parms)
}
probe process("/ddtv/tracedrv/test/dhg/hello/hello").function("*").return
{
        printf("USER %s %s %s\n", thread_indent(-1), probefunc(), $$return)
}
probe end
{
        printf ("end of tracing\n")
}

It only able to print "end of tracing", nothing in the hello is
printed. Attached is the binary of traceauto.ko.

BTW, it can trace kernel level code. Just not lucky on user-level code.

Any help is great appreciated.

Thanks,
Zhiwei

[-- Attachment #2: traceauto.ko --]
[-- Type: application/octet-stream, Size: 302835 bytes --]

Re: cannot trace user-level code in cross compilation environment

[Adrien Kunysz]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

On Fri, Apr 01, 2011 at 05:05:17PM +0800, Zhiwei Ying wrote:
> I built a systemtap1.4 on ubuntu 10.4 32 bits. And also cross-compile
> staprun to a develop board with kernel 2.6.35 x86.

Userland instrumentation requires the uprobe kernel patch. Uprobe is
not in the mainline kernel nor in Ubuntu so it seems normal you cannot
probe userland programs with these kernels. I believe the uprobes patch
is being maintained in this git repository:
http://git.kernel.org/?p=linux/kernel/git/srikar/linux-uprobes.git;a=tree

> On the host pc side,
> 
> stap -r /target-board/kernel xxx.stp -p 4 -m traceauto.ko
> 
> then copy the traceauto.ko to the board,
> staprun -u -v -m traceauto.ko -c ./hello
> 
> 
> xxx.stp
> probe process("/ddtv/tracedrv/test/dhg/hello/hello").function("*").call
> {
>         printf("USER, %s %s %s\n", thread_indent(1), pp(), $$parms)
> }
> probe process("/ddtv/tracedrv/test/dhg/hello/hello").function("*").return
> {
>         printf("USER %s %s %s\n", thread_indent(-1), probefunc(), $$return)
> }
> probe end
> {
>         printf ("end of tracing\n")
> }
> 
> It only able to print "end of tracing", nothing in the hello is
> printed. Attached is the binary of traceauto.ko.
> 
> BTW, it can trace kernel level code. Just not lucky on user-level code.

If it's still not working with uprobe, the output generated at
module-compilation time with stap -vvvvv would help to understand what
is going on.

> Any help is great appreciated.
> 
> Thanks,
> Zhiwei

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: cannot trace user-level code in cross compilation environment

[Zhiwei Ying]

I have patched the kernel with utrace and other systemtap requirement.
If there is no utrace support in the kernel, the staprun cannot load
the utrace.ko.

How can I trace the staprun? I turned on -v for staprun, but there is
not too much information provided.

On Fri, Apr 1, 2011 at 6:59 PM, Adrien Kunysz <adrien@kunysz.be> wrote:
> On Fri, Apr 01, 2011 at 05:05:17PM +0800, Zhiwei Ying wrote:
>> I built a systemtap1.4 on ubuntu 10.4 32 bits. And also cross-compile
>> staprun to a develop board with kernel 2.6.35 x86.
>
> Userland instrumentation requires the uprobe kernel patch. Uprobe is
> not in the mainline kernel nor in Ubuntu so it seems normal you cannot
> probe userland programs with these kernels. I believe the uprobes patch
> is being maintained in this git repository:
> http://git.kernel.org/?p=linux/kernel/git/srikar/linux-uprobes.git;a=tree
>
>> On the host pc side,
>>
>> stap -r /target-board/kernel xxx.stp -p 4 -m traceauto.ko
>>
>> then copy the traceauto.ko to the board,
>> staprun -u -v -m traceauto.ko -c ./hello
>>
>>
>> xxx.stp
>> probe process("/ddtv/tracedrv/test/dhg/hello/hello").function("*").call
>> {
>>         printf("USER, %s %s %s\n", thread_indent(1), pp(), $$parms)
>> }
>> probe process("/ddtv/tracedrv/test/dhg/hello/hello").function("*").return
>> {
>>         printf("USER %s %s %s\n", thread_indent(-1), probefunc(), $$return)
>> }
>> probe end
>> {
>>         printf ("end of tracing\n")
>> }
>>
>> It only able to print "end of tracing", nothing in the hello is
>> printed. Attached is the binary of traceauto.ko.
>>
>> BTW, it can trace kernel level code. Just not lucky on user-level code.
>
> If it's still not working with uprobe, the output generated at
> module-compilation time with stap -vvvvv would help to understand what
> is going on.
>
>> Any help is great appreciated.
>>
>> Thanks,
>> Zhiwei
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAk2VsAcACgkQKLX03ZSPZGy8rACfXElmGea5ijQb0qRhvtLGAlT3
> WEwAniUimy6j1/4V95ucSUTsmbGvF7Qf
> =rlJ2
> -----END PGP SIGNATURE-----
>
>

Re: cannot trace user-level code in cross compilation environment

[Josh Stone]

On 04/01/2011 02:05 AM, Zhiwei Ying wrote:
> Hi,
> 
> I built a systemtap1.4 on ubuntu 10.4 32 bits. And also cross-compile
> staprun to a develop board with kernel 2.6.35 x86.
> 
> On the host pc side,
> 
> stap -r /target-board/kernel xxx.stp -p 4 -m traceauto.ko
> 
> then copy the traceauto.ko to the board,
> staprun -u -v -m traceauto.ko -c ./hello
> 
> 
> xxx.stp
> probe process("/ddtv/tracedrv/test/dhg/hello/hello").function("*").call

Is your "hello" binary at the same path on the host and target?  The
runtime looks for probe targets based on the path, so it will never
notice if the application is started from a different path.

There's a bug open to address this, but for now the only workaround is
to copy or link the binary to matching locations on the host and target.
http://sourceware.org/bugzilla/show_bug.cgi?id=12331

Josh

Re: cannot trace user-level code in cross compilation environment

[Zhiwei Ying]

Yes, the binary in in the same path. if the binary is different,
staprun will complain "build-id mismatch"

Here is the detail for my cross-compilation:
cd systemtap-1.4
mkdir runtimebuild
cd runtimebuild
../runtime/staprun/configure --host=/crosstarget --prefix=
make
make install DESTDIR=/xxx

cd runtime/uprobes
#change KDIR in Makefile to crosstarget
make

cp staprun, stapio and uprobes to the board

on the board
# staprun -v traceauto.ko -b 3 -u -c ./hello.
stapio:main:33 Using a buffer of 3 MB.
stapio:start_cmd:200 block briefly
stapio:stp_main_loop:636 detaching pid 2289
stapio:start_cmd:219 execing target_cmd ./hello
hello world
stapio:cleanup_and_exit:403 detach=0
end of tracing
stapio:cleanup_and_exit:420 closing control channel
staprun:remove_module:213 Module traceauto removed

Any help is great appreciated!

Thanks,
Zhiwei

On Sat, Apr 2, 2011 at 2:25 AM, Josh Stone <jistone@redhat.com> wrote:
> On 04/01/2011 02:05 AM, Zhiwei Ying wrote:
>> Hi,
>>
>> I built a systemtap1.4 on ubuntu 10.4 32 bits. And also cross-compile
>> staprun to a develop board with kernel 2.6.35 x86.
>>
>> On the host pc side,
>>
>> stap -r /target-board/kernel xxx.stp -p 4 -m traceauto.ko
>>
>> then copy the traceauto.ko to the board,
>> staprun -u -v -m traceauto.ko -c ./hello
>>
>>
>> xxx.stp
>> probe process("/ddtv/tracedrv/test/dhg/hello/hello").function("*").call
>
> Is your "hello" binary at the same path on the host and target?  The
> runtime looks for probe targets based on the path, so it will never
> notice if the application is started from a different path.
>
> There's a bug open to address this, but for now the only workaround is
> to copy or link the binary to matching locations on the host and target.
> http://sourceware.org/bugzilla/show_bug.cgi?id=12331
>
> Josh
>

Re: cannot trace user-level code in cross compilation environment

[Josh Stone]

On 04/01/2011 05:52 PM, Zhiwei Ying wrote:
> Yes, the binary in in the same path. if the binary is different,
> staprun will complain "build-id mismatch"
> 
> Here is the detail for my cross-compilation:
> cd systemtap-1.4
> mkdir runtimebuild
> cd runtimebuild
> ../runtime/staprun/configure --host=/crosstarget --prefix=
> make
> make install DESTDIR=/xxx
> 
> cd runtime/uprobes
> #change KDIR in Makefile to crosstarget
> make
> 
> cp staprun, stapio and uprobes to the board
> 
> on the board
> # staprun -v traceauto.ko -b 3 -u -c ./hello.
> stapio:main:33 Using a buffer of 3 MB.
> stapio:start_cmd:200 block briefly
> stapio:stp_main_loop:636 detaching pid 2289
> stapio:start_cmd:219 execing target_cmd ./hello
> hello world
> stapio:cleanup_and_exit:403 detach=0
> end of tracing
> stapio:cleanup_and_exit:420 closing control channel
> staprun:remove_module:213 Module traceauto removed
> 
> Any help is great appreciated!

Please try these three options for stap, when compiling the ko:
  -DDEBUG_UPROBES -DDEBUG_TASK_FINDER -DDEBUG_TASK_FINDER_VMA

This should give much more debugging output to find the problem.

Josh

Re: cannot trace user-level code in cross compilation environment

[Zhiwei Ying]

Hi Josh,

You are correct. The problem is caused that two files are not in the same path.
I used a soft link to the binary on the host pc side, so in the script
they looks in the same directory, but actually they are not. Systemtap
will expand the softlink into real directory.

Thanks again,
Zhiwei

On Tue, Apr 5, 2011 at 6:56 AM, Josh Stone <jistone@redhat.com> wrote:
> On 04/01/2011 05:52 PM, Zhiwei Ying wrote:
>> Yes, the binary in in the same path. if the binary is different,
>> staprun will complain "build-id mismatch"
>>
>> Here is the detail for my cross-compilation:
>> cd systemtap-1.4
>> mkdir runtimebuild
>> cd runtimebuild
>> ../runtime/staprun/configure --host=/crosstarget --prefix=
>> make
>> make install DESTDIR=/xxx
>>
>> cd runtime/uprobes
>> #change KDIR in Makefile to crosstarget
>> make
>>
>> cp staprun, stapio and uprobes to the board
>>
>> on the board
>> # staprun -v traceauto.ko -b 3 -u -c ./hello.
>> stapio:main:33 Using a buffer of 3 MB.
>> stapio:start_cmd:200 block briefly
>> stapio:stp_main_loop:636 detaching pid 2289
>> stapio:start_cmd:219 execing target_cmd ./hello
>> hello world
>> stapio:cleanup_and_exit:403 detach=0
>> end of tracing
>> stapio:cleanup_and_exit:420 closing control channel
>> staprun:remove_module:213 Module traceauto removed
>>
>> Any help is great appreciated!
>
> Please try these three options for stap, when compiling the ko:
>  -DDEBUG_UPROBES -DDEBUG_TASK_FINDER -DDEBUG_TASK_FINDER_VMA
>
> This should give much more debugging output to find the problem.
>
> Josh
>