Unknown symbol error using stap in RHEL 5.4

Unknown symbol error using stap in RHEL 5.4

[Maynard Johnson]

Hi,
I maintain a package called stapitrace which has been in Fedora for some time.  Then late last year, we also put it in EPEL.  At that time, I did a fresh install of RHEL 5.4 on a ppc64 system and installed my package from the EPEL repository.  I tested it, and everything worked fine.  But it was brought to my attention recently that the package appeared broken on a user's RHEL 5.4 system.  When I started to debug the problem, I noted it was failing with "unknown symbol" errors, and dmesg showed the following:

stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol register_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol register_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol unregister_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unregister_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol register_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol register_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol unregister_uretprobe
stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unregister_uretprobe

====================

This looked suspiciously like the problem reported in https://bugzilla.redhat.com/show_bug.cgi?id=499677.  But I tried 'rmmod uprobes', "make -C /usr/share/systemtap/runtime/uprobes clean", and "rm -rf ~/.systemtap", but was not able to clear the problem.  I noted that the systemtap level was 1.1-3, which is different from a fresh RHEL 5.4 install.  So I tried uninstalling systemtap and installing the 0.9.7-5 rpm from the install DVD, but that didn't clear the error either.  As a sanity check, I did a fresh install of RHEL 5.4 on another system, and was able to use stapitrace with no problem there.

Any thoughts on what might be causing this problem?

Thanks!
-Maynard

Re: Unknown symbol error using stap in RHEL 5.4

[Jim Keniston]

On Wed, 2010-05-12 at 17:48 -0500, Maynard Johnson wrote:
> Hi,
> I maintain a package called stapitrace which has been in Fedora for some time.  Then late last year, we also put it in EPEL.  At that time, I did a fresh install of RHEL 5.4 on a ppc64 system and installed my package from the EPEL repository.  I tested it, and everything worked fine.  But it was brought to my attention recently that the package appeared broken on a user's RHEL 5.4 system.  When I started to debug the problem, I noted it was failing with "unknown symbol" errors, and dmesg showed the following:
> 
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol register_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol register_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol unregister_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unregister_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol register_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol register_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol unregister_uretprobe
> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unregister_uretprobe
> 
> ====================
> 
> This looked suspiciously like the problem reported in https://bugzilla.redhat.com/show_bug.cgi?id=499677.  But I tried 'rmmod uprobes', "make -C /usr/share/systemtap/runtime/uprobes clean", and "rm -rf ~/.systemtap", but was not able to clear the problem.  I noted that the systemtap level was 1.1-3, which is different from a fresh RHEL 5.4 install.  So I tried uninstalling systemtap and installing the 0.9.7-5 rpm from the install DVD, but that didn't clear the error either.  As a sanity check, I did a fresh install of RHEL 5.4 on another system, and was able to use stapitrace with no problem there.
> 
> Any thoughts on what might be causing this problem?

It appears that the source from which the kernel was built includes some
non-SystemTap version of uprobes, and that CONFIG_UPROBES is y or m in
the .config file.  If that's the case, stap sees that there are uprobes
functions in the kernel's Module.symvers file and uses the "native"
version of uprobes (running modprobe uprobes, if appropriate) instead of
its own version.  Unfortunately, functions like unmap_u[ret]probe, which
exist in SystemTap's version of uprobes, don't exist in many "native"
versions.

If the above is your problem, then I think the best recourse is to
rebuild the kernel without uprobes, so that SystemTap's version will get
used.

> 
> Thanks!
> -Maynard
> 

Jim

Re: Unknown symbol error using stap in RHEL 5.4

[Maynard Johnson]

Jim Keniston wrote:
> On Wed, 2010-05-12 at 17:48 -0500, Maynard Johnson wrote:
>> Hi,
>> I maintain a package called stapitrace which has been in Fedora for some time.  Then late last year, we also put it in EPEL.  At that time, I did a fresh install of RHEL 5.4 on a ppc64 system and installed my package from the EPEL repository.  I tested it, and everything worked fine.  But it was brought to my attention recently that the package appeared broken on a user's RHEL 5.4 system.  When I started to debug the problem, I noted it was failing with "unknown symbol" errors, and dmesg showed the following:
>>
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol register_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol register_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol unregister_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unregister_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unmap_uprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol register_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol register_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: disagrees about version of symbol unregister_uretprobe
>> stap_6e2a90fc5508789a23a65a03a12b9273_46646: Unknown symbol unregister_uretprobe
>>
>> ====================
>>
>> This looked suspiciously like the problem reported in https://bugzilla.redhat.com/show_bug.cgi?id=499677.  But I tried 'rmmod uprobes', "make -C /usr/share/systemtap/runtime/uprobes clean", and "rm -rf ~/.systemtap", but was not able to clear the problem.  I noted that the systemtap level was 1.1-3, which is different from a fresh RHEL 5.4 install.  So I tried uninstalling systemtap and installing the 0.9.7-5 rpm from the install DVD, but that didn't clear the error either.  As a sanity check, I did a fresh install of RHEL 5.4 on another system, and was able to use stapitrace with no problem there.
>>
>> Any thoughts on what might be causing this problem?
> 
> It appears that the source from which the kernel was built includes some
> non-SystemTap version of uprobes, and that CONFIG_UPROBES is y or m in
> the .config file.  If that's the case, stap sees that there are uprobes
> functions in the kernel's Module.symvers file and uses the "native"
> version of uprobes (running modprobe uprobes, if appropriate) instead of
> its own version.  Unfortunately, functions like unmap_u[ret]probe, which
> exist in SystemTap's version of uprobes, don't exist in many "native"
> versions.
> 
> If the above is your problem, then I think the best recourse is to
> rebuild the kernel without uprobes, so that SystemTap's version will get
> used.
Jim, thanks for your reply, but this is not really an option for users.

-Maynard
> 
>>
>> Thanks!
>> -Maynard
>>
> 
> Jim
> 
> 

Re: Unknown symbol error using stap in RHEL 5.4

[Josh Stone]

On 05/12/2010 03:48 PM, Maynard Johnson wrote:
> This looked suspiciously like the problem reported in
> https://bugzilla.redhat.com/show_bug.cgi?id=499677.  But I tried
> 'rmmod uprobes', "make -C /usr/share/systemtap/runtime/uprobes
> clean", and "rm -rf ~/.systemtap", but was not able to clear the
> problem.  I noted that the systemtap level was 1.1-3, which is
> different from a fresh RHEL 5.4 install.  So I tried uninstalling
> systemtap and installing the 0.9.7-5 rpm from the install DVD, but
> that didn't clear the error either.  As a sanity check, I did a fresh
> install of RHEL 5.4 on another system, and was able to use stapitrace
> with no problem there.
> 
> Any thoughts on what might be causing this problem?

Can you search the filesystem to make sure there's not any other version
of uprobes hanging around somewhere?

>> If the above is your problem, then I think the best recourse is to
>> rebuild the kernel without uprobes, so that SystemTap's version will get
>> used.
> Jim, thanks for your reply, but this is not really an option for users.

What Jim describes would only be the case if the user was already
compiling a custom kernel with uprobes in it.  The RHEL 5.4 kernel does
not have uprobes itself.

What is the kernel version on the problem system?


Josh

Re: Unknown symbol error using stap in RHEL 5.4

[Maynard Johnson]

Josh Stone wrote:
> On 05/12/2010 03:48 PM, Maynard Johnson wrote:
>> This looked suspiciously like the problem reported in
>> https://bugzilla.redhat.com/show_bug.cgi?id=499677.  But I tried
>> 'rmmod uprobes', "make -C /usr/share/systemtap/runtime/uprobes
>> clean", and "rm -rf ~/.systemtap", but was not able to clear the
>> problem.  I noted that the systemtap level was 1.1-3, which is
>> different from a fresh RHEL 5.4 install.  So I tried uninstalling
>> systemtap and installing the 0.9.7-5 rpm from the install DVD, but
>> that didn't clear the error either.  As a sanity check, I did a fresh
>> install of RHEL 5.4 on another system, and was able to use stapitrace
>> with no problem there.
>>
>> Any thoughts on what might be causing this problem?
> 
> Can you search the filesystem to make sure there's not any other version
> of uprobes hanging around somewhere?

Josh, thanks for that tip.  The system was booted on 2.6.18-164.11.1.el5.  However, I did find a uprobes.ko in some /lib/modules sub-directory that was something other than 2.6.18-164.11.1.el5.  Looking at the system in more detail, I found several old kernel rpms and some other pieces that looked like someone had at one time, built a custom kernel.  I think the system was majorly confused.  :-) I cleaned everything up, updated the kernel to 2.6.18-194.3.1.el5 and rebooted.  For some reason (that I can't remember now), I also updated systemtap (and runtime) from 0.9.7 to 1.1-3.  After making one tweak in /usr/share/systemtap/tapsets, the stapitrace tool worked!  Thanks.

*Frank*, the tweak I mentioned above was this:  The stapitrace rpm installation adds a tapset script to /usr/share/systemtap/tapsets/ppc64.  I found that with the updated systemtap, I needed to move that file to /usr/share/systemtap/tapsets/powerpc.  I didn't know about this change in systemtap directory structure before now.  What was the reason for the change?  What would you recommend as the best solution to this problem?

Thanks.

-Maynard

> 
>>> If the above is your problem, then I think the best recourse is to
>>> rebuild the kernel without uprobes, so that SystemTap's version will get
>>> used.
>> Jim, thanks for your reply, but this is not really an option for users.
> 
> What Jim describes would only be the case if the user was already
> compiling a custom kernel with uprobes in it.  The RHEL 5.4 kernel does
> not have uprobes itself.
> 
> What is the kernel version on the problem system?
> 
> 
> Josh

Re: Unknown symbol error using stap in RHEL 5.4

[Frank Ch. Eigler]

Maynard Johnson <maynardj@us.ibm.com> writes:

> [...]  *Frank*, the tweak I mentioned above was this: The stapitrace
> rpm installation adds a tapset script to
> /usr/share/systemtap/tapsets/ppc64.  I found that with the updated
> systemtap, I needed to move that file to
> /usr/share/systemtap/tapsets/powerpc.  [...]

This was a consequence of PR4186, a change noted in NEWS for release 1.0.

  The systemtap notion of "architecture" now matches the kernel's, rather
  than that of "uname -m".  This means that 32-bit i386 family are all
  known as "i386" rather than "i386" or "i686"; "ppc64" as "powerpc";
  "s390x" as "s390", and so on.  This is consistent between the new
  "-a ARCH" flag and the script-level %( arch ... %) conditional.

If you don't want to move stapitrace over to the new conventions, you
could put your tapset file under /usr/share/systemtap/tapset directly,
and guard it with  %( arch == "powerpc" || arch == "ppc64" %? .... %)


- FChE

Re: Unknown symbol error using stap in RHEL 5.4

[Josh Stone]

On 05/13/2010 11:07 AM, Frank Ch. Eigler wrote:
> Maynard Johnson <maynardj@us.ibm.com> writes:
> 
>> [...]  *Frank*, the tweak I mentioned above was this: The stapitrace
>> rpm installation adds a tapset script to
>> /usr/share/systemtap/tapsets/ppc64.  I found that with the updated
>> systemtap, I needed to move that file to
>> /usr/share/systemtap/tapsets/powerpc.  [...]
> 
> This was a consequence of PR4186, a change noted in NEWS for release 1.0.

> If you don't want to move stapitrace over to the new conventions, you
> could put your tapset file under /usr/share/systemtap/tapset directly,
> and guard it with  %( arch == "powerpc" || arch == "ppc64" %? .... %)

Well, that preprocessor-|| was only introduced in 1.0 as well. :(

I think the easiest thing is to just move the tapset to the new location
and rpm-Require systemtap >= 1.0.

Josh