accessing target globals with systemtap

accessing target globals with systemtap

[Jeff Haran]

Hi,

After reading the documentation, I assumed the following would result in the display of the address of the kernel global named arp_tbl:

[root@s01b06 jharan]# cat temp.stp
probe begin
{
        printf("arp_tbl %p\n", @var("arp_tbl@net/ipv4/arp.c"))
}

But when I try to compile it, I get this:

[root@s01b06 jharan]# stap -v -r 2.6.32-220.el6.x86_64 temp.stp -m temp -p 4
parse error: unknown statistic operator @var
        saw: identifier '@var' at temp.stp:3:25
     source:    printf("arp_tbl %p\n", @var("arp_tbl@net/ipv4/arp.c"))
                                       ^
parse error: expected statement
        saw: temp.stp EOF
2 parse errors.
Pass 1: parsed user script and 77 library script(s) using 97284virt/22908res/2984shr kb, in 140usr/0sys/144real ms.
Pass 1: parse failed.  Try again with another '--vp 1' option.
[root@s01b06 jharan]#

I am guessing that's because of my older version of stap:

[root@s01b06 jharan]# stap --version
Systemtap translator/driver (version 1.6/0.152 non-git sources)
Copyright (C) 2005-2011 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBRPM LIBSQLITE3 NSS BOOST_SHARED_PTR TR1_UNORDERED_MAP NLS
[root@s01b06 jharan]#

Is there a way for me to get access to a kernel global in my version of stap?
Or do I need to upgrade in order to do this?
Or am I missing something more basic?

Thanks,

Jeff Haran

Re: accessing target globals with systemtap

[Frank Ch. Eigler]

Jeff Haran <Jeff.Haran@citrix.com> writes:

> [...]
> probe begin
> {
>         printf("arp_tbl %p\n", @var("arp_tbl@net/ipv4/arp.c"))
> }

This won't work for a few more reasons.

@var() resolves within the context of the probe point.  probe begin
does not offer a useful context -- it doesn't say e.g. whether this
is a kernel or userspace program whose "arp_tbl" you're looking for.
Try instead printing this value from a kernel.function type probe.

Second, @var("arp_tbl") will result in a struct neigh_table, something
that cannot be printed with a %p.  Systemtap will give you a type
mismatch error, unless e.g. you print  & @var(...)  to print the address.
(You can hex-dump the structure via printf("%*m") etc.)

> [...]
> I am guessing that's because of my older version of stap:

That's correct.

> [root@s01b06 jharan]# stap --version
> Systemtap translator/driver (version 1.6/0.152 non-git sources)
> [...]

> Is there a way for me to get access to a kernel global in my version of stap?
> Or do I need to upgrade in order to do this?

Another way would be to use embedded-C:

%{ 
#include <net/arp.h>
%}
function the_table_address:long () %{ /* unmangled */
  /* rely on EXPORT_SYMBOL ... to let this resolve */
  THIS->__retvalue = & arp_tbl;
%}
probe begin {
  // to find the base address
  printf("%p\n", the_table_address())
  // to fetch fields:
  printf("%d\n", @cast(the_table_address(),"neigh_table","kernel")->family);
}


> Or am I missing something more basic?

Nope.

- FChE

RE: accessing target globals with systemtap

[Jeff Haran]

> -----Original Message-----
> From: Frank Ch. Eigler [mailto:fche@redhat.com]
> Sent: Tuesday, September 11, 2012 5:16 PM
> To: Jeff Haran
> Cc: 'systemtap@sourceware.org'
> Subject: Re: accessing target globals with systemtap
> 
...
> Try instead printing this value from a kernel.function type probe.

Frank,

Thanks a bunch for helping me through this. I did try another kernel.function type probe after reading some related email. I figured arp_init() in net/ipv4/arp.c would have arp_tbl in its scope and it should get called before the functions I am interesting in having the value of arp_tbl in:

171 struct neigh_table arp_tbl = {
...
1235 void __init arp_init(void)
1236 {
1237         neigh_table_init(&arp_tbl);
1238
1239         dev_add_pack(&arp_packet_type);
1240         arp_proc_init();
1241 #ifdef CONFIG_SYSCTL
1242         neigh_sysctl_register(NULL, &arp_tbl.parms, NET_IPV4,
1243                               NET_IPV4_NEIGH, "ipv4", NULL, NULL);
1244 #endif
1245         register_netdevice_notifier(&arp_netdev_notifier);
1246 }

But that yields this:

[root@s01b06 jharan]# cat temp2.stp
global arp_tbl

probe kernel.function("arp_init").call
{
        arp_tbl = &$arp_tbl
}

[root@s01b06 jharan]# stap -v -r 2.6.32-220.el6.x86_64 temp2.stp -m temp2 -p 4
Pass 1: parsed user script and 77 library script(s) using 97284virt/22832res/2912shr kb, in 130usr/10sys/143real ms.
semantic error: no match while resolving probe point kernel.function("arp_init").call
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 1 global(s) using 249284virt/122356res/79740shr kb, in 410usr/70sys/484real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.
[root@s01b06 jharan]#

I also tried this (leaving out  the '&'), just to see if that would change what it was complaining about:

[root@s01b06 jharan]# cat temp2.stp
global arp_tbl

probe kernel.function("arp_init").call
{
        arp_tbl = $arp_tbl
}

[root@s01b06 jharan]# stap -v -r 2.6.32-220.el6.x86_64 temp2.stp -m temp2 -p 4
Pass 1: parsed user script and 77 library script(s) using 97284virt/22836res/2912shr kb, in 120usr/10sys/139real ms.
semantic error: no match while resolving probe point kernel.function("arp_init").call
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 1 global(s) using 249288virt/122360res/79740shr kb, in 400usr/70sys/463real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.
[root@s01b06 jharan]#

From the error messages, it seems like it's having problems with finding arp_init() rather than any particular problem with the probe body.

Is this because arp_init is declared with __init?

I also tried kernel.statement instead:

[root@s01b06 jharan]# cat temp3.stp
global arp_tbl

probe kernel.statement("*@net/ipv4/arp.c:1238")
{
        arp_tbl = $arp_tbl
}
[root@s01b06 jharan]# stap -v -r 2.6.32-220.el6.x86_64 temp3.stp -m temp3 -p 4
Pass 1: parsed user script and 77 library script(s) using 97284virt/22832res/2912shr kb, in 130usr/0sys/139real ms.
semantic error: no match while resolving probe point kernel.statement("*@net/ipv4/arp.c:1238")
Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 1 global(s) using 267124virt/76472res/17420shr kb, in 170usr/80sys/238real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.
[root@s01b06 jharan]#

I think I have that kernel.statement syntax right. This one did work as expected and it looks pretty much the same:

probe kernel.statement("*@net/core/neighbour.c:934")
...

It seems like it just can't find that arp_init code, regardless of the probe body.

Is there something I need to do in order to roll kernel.function or kernel.statement in order to probe arp_init()?

Also, what is the recommended way of getting the address of an in-scope structure instance?

As you've pointed out this is wrong:

        arp_tbl = $arp_tbl

Is this the way to do it?

        arp_tbl = &$arp_tbl

> > Is there a way for me to get access to a kernel global in my version of stap?
> 
> Another way would be to use embedded-C:
> 
> %{
> #include <net/arp.h>
> %}
> function the_table_address:long () %{ /* unmangled */
>   /* rely on EXPORT_SYMBOL ... to let this resolve */
>   THIS->__retvalue = & arp_tbl;
> %}
> probe begin {
>   // to find the base address
>   printf("%p\n", the_table_address())
>   // to fetch fields:
>   printf("%d\n", @cast(the_table_address(),"neigh_table","kernel")-
> >family);
> }

I'll give this embedded C a try.

Thanks again,

Jeff Haran

RE: accessing target globals with systemtap

[Jeff Haran]

> -----Original Message-----
> From: systemtap-owner@sourceware.org [mailto:systemtap-
> owner@sourceware.org] On Behalf Of Jeff Haran
> Sent: Tuesday, September 11, 2012 6:00 PM
> To: 'Frank Ch. Eigler'
> Cc: 'systemtap@sourceware.org'
> Subject: RE: accessing target globals with systemtap
... 
> > > Is there a way for me to get access to a kernel global in my version of
> stap?
> >
> > Another way would be to use embedded-C:
> >
> > %{
> > #include <net/arp.h>
> > %}
> > function the_table_address:long () %{ /* unmangled */
> >   /* rely on EXPORT_SYMBOL ... to let this resolve */
> >   THIS->__retvalue = & arp_tbl;
> > %}
> > probe begin {
> >   // to find the base address
> >   printf("%p\n", the_table_address())
> >   // to fetch fields:
> >   printf("%d\n", @cast(the_table_address(),"neigh_table","kernel")-
> > >family);
> > }
> 
> I'll give this embedded C a try.

Just an FYI for posterity. The above generated a warning in the C compilation pass, which since warnings are treated as errors meant no .ko file was produced.

But replacing this:

	THIS->__retvalue = & arp_tbl;

with this:

	THIS->__retvalue = (int64_t) & arp_tbl;

built fine and it seems to have worked right too:

[root@s01b05 jharan]# staprun temp4.ko
0xffffffff81b157c0
2
^C[root@s01b05 jharan]#

Thanks again,

Jeff Haran

Re: accessing target globals with systemtap

[Frank Ch. Eigler]

Hi -

On Tue, Sep 11, 2012 at 05:59:58PM -0700, Jeff Haran wrote:
> [...]
> 1235 void __init arp_init(void)
> [...]

It does look like a tasty little tidbit.  Unfortunately, __init
functions in the kernel are normally tossed into the memory hole right
after bootup, so are not available for probing by the time we proles
come along.  With a "stap --vp 02" option added, you'll get a hint at
the problem:

probe arp_init@net/ipv4/arp.c:1276 kernel reloc=.dynamic pc=0xffffffff81bb7f6f init/exit - skipped


> [...]
> Also, what is the recommended way of getting the address of an in-scope structure instance?
> 
> As you've pointed out this is wrong:
>         arp_tbl = $arp_tbl
> Is this the way to do it?
>         arp_tbl = &$arp_tbl

Yes.



- FChE