about probe libc

about probe libc

[Huang jw]

Hi all,
       I'm trying to probe memcpy from libc. I have installed
debuginfo for libc(glibc-debuginfo-2.5-42.x86_64.rpm), but when I use
the follow systemtap instruction, it reoprted an error.
Instruction list I used:
sudo stap -e 'probe
process("/lib/libc.so.6").function("memcpy").return{println(probfunc())}'
-c 'ls'

Error information:
"
WARNING: cannot find module /lib/libc-2.5.so debuginfo: No DWARF
information found
semantic error: no match while resolving probe point
process("/lib/libc.so.6").function("memcpy").return"

The os I used is REL5.4, libc version is 2.5.42.
Does any one know why?

Re: about probe libc

[Srikar Dronamraju]

> Hi all,
>        I'm trying to probe memcpy from libc. I have installed
> debuginfo for libc(glibc-debuginfo-2.5-42.x86_64.rpm), but when I use
> the follow systemtap instruction, it reoprted an error.

Right you have installed debuginfo for the 64 bit version.

> Instruction list I used:
> sudo stap -e 'probe
> process("/lib/libc.so.6").function("memcpy").return{println(probfunc())}'
> -c 'ls'
> 
> Error information:
> "
> WARNING: cannot find module /lib/libc-2.5.so debuginfo: No DWARF
> information found
> semantic error: no match while resolving probe point
> process("/lib/libc.so.6").function("memcpy").return"

ls on x86_64 is a 64 bit binary that links with /lib64/libc.so.
So you were right in installing glibc-debuginfo-2.5-42.x86_64.rpm.

However can you try using 
process("/lib64/libc.so.6").function("memcpy").return"

--
Thanks and Regards
Srikar

Re: about probe libc

[Huang jw]

2010/8/10 Srikar Dronamraju <srikar@linux.vnet.ibm.com>:
>> Hi all,
>>        I'm trying to probe memcpy from libc. I have installed
>> debuginfo for libc(glibc-debuginfo-2.5-42.x86_64.rpm), but when I use
>> the follow systemtap instruction, it reoprted an error.
>
> Right you have installed debuginfo for the 64 bit version.
>
>> Instruction list I used:
>> sudo stap -e 'probe
>> process("/lib/libc.so.6").function("memcpy").return{println(probfunc())}'
>> -c 'ls'
>>
>> Error information:
>> "
>> WARNING: cannot find module /lib/libc-2.5.so debuginfo: No DWARF
>> information found
>> semantic error: no match while resolving probe point
>> process("/lib/libc.so.6").function("memcpy").return"
>
> ls on x86_64 is a 64 bit binary that links with /lib64/libc.so.
> So you were right in installing glibc-debuginfo-2.5-42.x86_64.rpm.
>
> However can you try using
> process("/lib64/libc.so.6").function("memcpy").return"
>

Thanks, I also. tried that, but it still didn't work.

semantic error: no match while resolving probe point
process("/lib64/libc-2.5.so").function("memcpy").return
semantic error: no probes found

Re: about probe libc

[Roland McGrath]

> Right you have installed debuginfo for the 64 bit version.

That's a good point!  I had not noticed that bit of pilot error.  FWIW, I
think you can install both the i?86 and x86_64 glibc-debuginfo rpms at the
same time.

But it turns out that was not only the problem anyway.  I actually
reproduced it on an i686 install of RHEL5.~5 and my explanation about the
CRC mismatch was based on that.


Thanks,
Roland

Re: about probe libc

[Roland McGrath]

The RHEL5 debuginfo for libc (perhaps for all .so's?) has a CRC mismatch
with the .gnu_debuglink section embedded in the binary.  This indicates a
problem in the rpm build procedure, but the binaries are already built.
To avoid the CRC checking you can use a SYSTEMTAP_DEBUGINFO_PATH setting
that begins with - instead of +, for example:

SYSTEMTAP_DEBUGINFO_PATH=-/usr/lib/debug  stap -e 'probe process("/lib/libc.so.6").function("memcpy").return{println(probefunc())}' -c 'ls'

worked for me where it before got the same failure mode you saw.
("Worked" just meaning doesn't get that "No DWARF" error any more.)

The systemtap default path is:
	+:.debug:/usr/lib/debug:/var/cache/abrt-di/usr/lib/debug:build
That leading + flag (vs -) enables the CRC checking by default.

When disabling this check, you always need to double-check by hand that you
really have the correct .debug files installed to match the binaries you
are using (i.e. from foo-N-V-R.A.rpm and foo-debuginfo-N-V-R-A.rpm with
exactly matching N-V-R.A), because you are no longer getting any kind of
automatic checking that they are an exact match.

On systems newer than RHEL5 (e.g. Fedora >= 8, RHEL6), all the binaries and
.debug files will have build IDs.  When binaries have build IDs, those are
a reliable verification that you have the right files installed, and so the
CRC check is never consulted (whether it's correct, or broken as for RHEL5
libc.so.debug).  The - flag in the debuginfo path setting has no effect on
build ID checking, so, if you like, you can use a uniform setting both for
RHEL5 and for systems that do have build IDs, and lose nothing in the newer
and better environments.

Note that in this example I didn't see any probe hits, and that may well be
"correct".  memcpy is one of several special-case functions that (in
optimized code) are almost always either defined as macros or inlines in
header files, or directly compiled away by the compiler as a special built-in.

You are unlikely to be able to get any reliable probing of memcpy (or some
other examples like other simple <string.h> functions), except perhaps for
the calls in your own code if it is compiled without optimization (-O0).


Thanks,
Roland

Re: about probe libc

[Huang jw]

Thanks for your suggestion. In my exp, I want to get the statistics of
memory activity of the whole system, such as memcpy, malloc, free. For
malloc and free, we can use hook. For memcpy, I tried ltrace, but
ltrace still has problem with pthread. So I choose systemtap.  As you
said, it's difficult to trace memcpy by systemtap because of inline or
Macros. Any more suggestions about memcpy tracing ?

2010/8/10 Roland McGrath <roland@redhat.com>:
> The RHEL5 debuginfo for libc (perhaps for all .so's?) has a CRC mismatch
> with the .gnu_debuglink section embedded in the binary.  This indicates a
> problem in the rpm build procedure, but the binaries are already built.
> To avoid the CRC checking you can use a SYSTEMTAP_DEBUGINFO_PATH setting
> that begins with - instead of +, for example:
>
> SYSTEMTAP_DEBUGINFO_PATH=-/usr/lib/debug  stap -e 'probe process("/lib/libc.so.6").function("memcpy").return{println(probefunc())}' -c 'ls'
>
> worked for me where it before got the same failure mode you saw.
> ("Worked" just meaning doesn't get that "No DWARF" error any more.)
>
> The systemtap default path is:
>        +:.debug:/usr/lib/debug:/var/cache/abrt-di/usr/lib/debug:build
> That leading + flag (vs -) enables the CRC checking by default.
>
> When disabling this check, you always need to double-check by hand that you
> really have the correct .debug files installed to match the binaries you
> are using (i.e. from foo-N-V-R.A.rpm and foo-debuginfo-N-V-R-A.rpm with
> exactly matching N-V-R.A), because you are no longer getting any kind of
> automatic checking that they are an exact match.
>
> On systems newer than RHEL5 (e.g. Fedora >= 8, RHEL6), all the binaries and
> .debug files will have build IDs.  When binaries have build IDs, those are
> a reliable verification that you have the right files installed, and so the
> CRC check is never consulted (whether it's correct, or broken as for RHEL5
> libc.so.debug).  The - flag in the debuginfo path setting has no effect on
> build ID checking, so, if you like, you can use a uniform setting both for
> RHEL5 and for systems that do have build IDs, and lose nothing in the newer
> and better environments.
>
> Note that in this example I didn't see any probe hits, and that may well be
> "correct".  memcpy is one of several special-case functions that (in
> optimized code) are almost always either defined as macros or inlines in
> header files, or directly compiled away by the compiler as a special built-in.
>
> You are unlikely to be able to get any reliable probing of memcpy (or some
> other examples like other simple <string.h> functions), except perhaps for
> the calls in your own code if it is compiled without optimization (-O0).
>
>
> Thanks,
> Roland
>

Re: about probe libc

[Frank Ch. Eigler]

Huang jw <huangjiangwei@gmail.com> writes:

> As you said, it's difficult to trace memcpy by systemtap because of
> inline or Macros. [...]

One more thing worth trying is probing the inlined instances within
the process binary (instead of just the copy in the libc.so), as
in

        probe process("a.out").function("memcpy"),
              process("/lib/libc.5.so").function("memcpy") {
              log($$parms)
        }

The former may catch the synthetic ones.

- FChE