More user-space probes

More user-space probes

[David Smith]

I've just finished implementing some new forms of user-space probing.
Here's an example script:

  probe process("/bin/ls").exec { print("ls exec'ed!\n") }
  probe process("/bin/ls").syscall { printf("|%d", $syscall) }
  probe process("/bin/ls").syscall.return { print("+") }
  probe process("/bin/ls").death { print("\nls done!\n") }

There are 5 new probe variants:

  process(PID_OR_PATH).clone
  process(PID_OR_PATH).exec
  process(PID_OR_PATH).death
  process(PID_OR_PATH).syscall
  process(PID_OR_PATH).syscall.return

(Internally, these are implemented using utrace.  These probe types also
use the new "task_finder" framework, which provides the support by
probing by PID or by PATH.  Eventually, the existing uprobes probes (and
the future itrace probes) should be converted to use the "task_finder"
framework.)

Feel free to try out the new probe types and let me know if you find any
issues.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: More user-space probes

[Frank Ch. Eigler]

David Smith <dsmith@redhat.com> writes:

> I've just finished implementing some new forms of user-space probing.

Fun.

> [...]  Eventually, [...] the future itrace probes should be
> converted to use the "task_finder" framework.)

Even more: the itrace stuff (bug #5949) should be a direct sibling to
these ".syscall"/".death"/.utrace-event probes, as far as possible.


- FChE

Re: More user-space probes

[Srinivasa D S]

On Tuesday 22 April 2008 07:10:32 pm David Smith wrote:
>> (Internally, these are implemented using utrace.  These probe types also
> use the new "task_finder" framework, which provides the support by
> probing by PID or by PATH.  Eventually, the existing uprobes probes (and
> the future itrace probes) should be converted to use the "task_finder"
> framework.)
>
> Feel free to try out the new probe types and let me know if you find any
> issues.


I hit a compilation error while testing new probes mentioned in the
mail on my 2.6.25-rc6 ppc system.

In file included from
/tmp/stapTlEH0O/stap_eaf6c06588419104dbe101b4d46dbe00_410.c:108:
/usr/local/share/systemtap/runtime/task_finder.c: In function
'__stp_get_mm_path':
/usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
argument 1 of 'd_path' from incompatible pointer type
/usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
argument 2 of 'd_path' from incompatible pointer type
/usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
argument 3 of 'd_path' makes integer from pointer without a cast
/usr/local/share/systemtap/runtime/task_finder.c:158: error: too many
arguments to function 'd_path'

With below fix, It worked for me, But Iam not sure about holding reference 
(through mntget(),dget()) dentry and mnt object while calling d_path.
Please let me know your comments on this.


---
 runtime/task_finder.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

Index: src/runtime/task_finder.c
===================================================================
--- src.orig/runtime/task_finder.c
+++ src/runtime/task_finder.c
@@ -153,11 +153,8 @@ __stp_get_mm_path(struct mm_struct *mm, 
 		vma = vma->vm_next;
 	}
 	if (vma) {
-		struct vfsmount *mnt = mntget(vma->vm_file->f_vfsmnt);
-		struct dentry *dentry = dget(vma->vm_file->f_dentry);
-		rc = d_path(dentry, mnt, buf, buflen);
-		dput(dentry);
-		mntput(mnt);
+		struct path p = vma->vm_file->f_path;
+		rc = d_path(&p, buf, buflen);
 	}
 	else {
 		*buf = '\0';

Re: More user-space probes

[Srinivasa D S]

On Tuesday 22 April 2008 07:10:32 pm David Smith wrote:
>> (Internally, these are implemented using utrace.  These probe types also
> use the new "task_finder" framework, which provides the support by
> probing by PID or by PATH.  Eventually, the existing uprobes probes (and
> the future itrace probes) should be converted to use the "task_finder"
> framework.)
>
> Feel free to try out the new probe types and let me know if you find any
> issues.


I hit a compilation error while testing new probes mentioned in the
mail on my 2.6.25-rc6 ppc system.

In file included from
/tmp/stapTlEH0O/stap_eaf6c06588419104dbe101b4d46dbe00_410.c:108:
/usr/local/share/systemtap/runtime/task_finder.c: In function
'__stp_get_mm_path':
/usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
argument 1 of 'd_path' from incompatible pointer type
/usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
argument 2 of 'd_path' from incompatible pointer type
/usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
argument 3 of 'd_path' makes integer from pointer without a cast
/usr/local/share/systemtap/runtime/task_finder.c:158: error: too many
arguments to function 'd_path'

With below fix, It worked for me, But Iam not sure about holding reference 
(through mntget(),dget()) dentry and mnt object while calling d_path.
Please let me know your comments on this.


---
 runtime/task_finder.c |    7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

Index: src/runtime/task_finder.c
===================================================================
--- src.orig/runtime/task_finder.c
+++ src/runtime/task_finder.c
@@ -153,11 +153,8 @@ __stp_get_mm_path(struct mm_struct *mm, 
 		vma = vma->vm_next;
 	}
 	if (vma) {
-		struct vfsmount *mnt = mntget(vma->vm_file->f_vfsmnt);
-		struct dentry *dentry = dget(vma->vm_file->f_dentry);
-		rc = d_path(dentry, mnt, buf, buflen);
-		dput(dentry);
-		mntput(mnt);
+		struct path p = vma->vm_file->f_path;
+		rc = d_path(&p, buf, buflen);
 	}
 	else {
 		*buf = '\0';

Re: More user-space probes

[David Smith]

Srinivasa D S wrote:
> On Tuesday 22 April 2008 07:10:32 pm David Smith wrote:
>>> (Internally, these are implemented using utrace.  These probe types also
>> use the new "task_finder" framework, which provides the support by
>> probing by PID or by PATH.  Eventually, the existing uprobes probes (and
>> the future itrace probes) should be converted to use the "task_finder"
>> framework.)
>>
>> Feel free to try out the new probe types and let me know if you find any
>> issues.
> 
> 
> I hit a compilation error while testing new probes mentioned in the
> mail on my 2.6.25-rc6 ppc system.
> 
> In file included from
> /tmp/stapTlEH0O/stap_eaf6c06588419104dbe101b4d46dbe00_410.c:108:
> /usr/local/share/systemtap/runtime/task_finder.c: In function
> '__stp_get_mm_path':
> /usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
> argument 1 of 'd_path' from incompatible pointer type
> /usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
> argument 2 of 'd_path' from incompatible pointer type
> /usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
> argument 3 of 'd_path' makes integer from pointer without a cast
> /usr/local/share/systemtap/runtime/task_finder.c:158: error: too many
> arguments to function 'd_path'
> 
> With below fix, It worked for me, But Iam not sure about holding reference 
> (through mntget(),dget()) dentry and mnt object while calling d_path.
> Please let me know your comments on this.
> 
> 
> ---
>  runtime/task_finder.c |    7 ++-----
>  1 file changed, 2 insertions(+), 5 deletions(-)
> 
> Index: src/runtime/task_finder.c
> ===================================================================
> --- src.orig/runtime/task_finder.c
> +++ src/runtime/task_finder.c
> @@ -153,11 +153,8 @@ __stp_get_mm_path(struct mm_struct *mm, 
>  		vma = vma->vm_next;
>  	}
>  	if (vma) {
> -		struct vfsmount *mnt = mntget(vma->vm_file->f_vfsmnt);
> -		struct dentry *dentry = dget(vma->vm_file->f_dentry);
> -		rc = d_path(dentry, mnt, buf, buflen);
> -		dput(dentry);
> -		mntput(mnt);
> +		struct path p = vma->vm_file->f_path;
> +		rc = d_path(&p, buf, buflen);
>  	}
>  	else {
>  		*buf = '\0';
> 

Here's what I just checked in.  Can you check and see if it works for you?

diff --git a/runtime/task_finder.c b/runtime/task_finder.c

index d0a9a40..ffbc3d0 100644

--- a/runtime/task_finder.c

+++ b/runtime/task_finder.c

@@ -153,11 +153,12 @@ __stp_get_mm_path(struct mm_struct *mm, char *buf,
int bu\
flen)

                vma = vma->vm_next;

        }

        if (vma) {

-               struct vfsmount *mnt = mntget(vma->vm_file->f_vfsmnt);

-               struct dentry *dentry = dget(vma->vm_file->f_dentry);

-               rc = d_path(dentry, mnt, buf, buflen);

-               dput(dentry);

-               mntput(mnt);

+#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,25)

+               rc = d_path(vma->vm_file->f_dentry,
vma->vm_file->f_vfsmnt,
+                           buf, buflen);

+#else

+               rc = d_path(vma->vm_file, buf, buflen);

+#endif

        }

        else {

                *buf = '\0';

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: More user-space probes

[Srinivasa D S]

On Thursday 24 April 2008 11:25:40 pm David Smith wrote:
> Srinivasa D S wrote:
> > On Tuesday 22 April 2008 07:10:32 pm David Smith wrote:
> >>> (Internally, these are implemented using utrace.  These probe types
> >>> also
> >>
> >> use the new "task_finder" framework, which provides the support by
> >> probing by PID or by PATH.  Eventually, the existing uprobes probes (and
> >> the future itrace probes) should be converted to use the "task_finder"
> >> framework.)
> >>
> >> Feel free to try out the new probe types and let me know if you find any
> >> issues.
> >
> > I hit a compilation error while testing new probes mentioned in the
> > mail on my 2.6.25-rc6 ppc system.
> >
> > In file included from
> > /tmp/stapTlEH0O/stap_eaf6c06588419104dbe101b4d46dbe00_410.c:108:
> > /usr/local/share/systemtap/runtime/task_finder.c: In function
> > '__stp_get_mm_path':
> > /usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
> > argument 1 of 'd_path' from incompatible pointer type
> > /usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
> > argument 2 of 'd_path' from incompatible pointer type
> > /usr/local/share/systemtap/runtime/task_finder.c:158: warning: passing
> > argument 3 of 'd_path' makes integer from pointer without a cast
> > /usr/local/share/systemtap/runtime/task_finder.c:158: error: too many
> > arguments to function 'd_path'
> >
> > With below fix, It worked for me, But Iam not sure about holding
> > reference (through mntget(),dget()) dentry and mnt object while calling
> > d_path. Please let me know your comments on this.
> >
> >
> > ---
> >  runtime/task_finder.c |    7 ++-----
> >  1 file changed, 2 insertions(+), 5 deletions(-)
> >
> > Index: src/runtime/task_finder.c
> > ===================================================================
> > --- src.orig/runtime/task_finder.c
> > +++ src/runtime/task_finder.c
> > @@ -153,11 +153,8 @@ __stp_get_mm_path(struct mm_struct *mm,
> >  		vma = vma->vm_next;
> >  	}
> >  	if (vma) {
> > -		struct vfsmount *mnt = mntget(vma->vm_file->f_vfsmnt);
> > -		struct dentry *dentry = dget(vma->vm_file->f_dentry);
> > -		rc = d_path(dentry, mnt, buf, buflen);
> > -		dput(dentry);
> > -		mntput(mnt);
> > +		struct path p = vma->vm_file->f_path;
> > +		rc = d_path(&p, buf, buflen);
> >  	}
> >  	else {
> >  		*buf = '\0';
>
> Here's what I just checked in.  Can you check and see if it works for you?
>

No, It didn't worked for me.  I did a small change to it and now it works.


---
 runtime/task_finder.c |   15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

Index: src/runtime/task_finder.c
===================================================================
--- src.orig/runtime/task_finder.c
+++ src/runtime/task_finder.c
@@ -153,11 +153,16 @@ __stp_get_mm_path(struct mm_struct *mm,
                vma = vma->vm_next;
        }
        if (vma) {
-               struct vfsmount *mnt = mntget(vma->vm_file->f_vfsmnt);
-               struct dentry *dentry = dget(vma->vm_file->f_dentry);
-               rc = d_path(dentry, mnt, buf, buflen);
-               dput(dentry);
-               mntput(mnt);
+               #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,25)
+
+               rc = d_path(vma->vm_file->f_dentry,
+                       vma->vm_file->f_vfsmnt, buf, buflen);
+
+               #else
+
+               rc = d_path(&(vma->vm_file->f_path), buf, buflen);
+
+               #endif
        }
        else {
                *buf = '\0';

Re: More user-space probes

[David Smith]

Srinivasa D S wrote:
> On Thursday 24 April 2008 11:25:40 pm David Smith wrote:
>> Srinivasa D S wrote:
>>> On Tuesday 22 April 2008 07:10:32 pm David Smith wrote:
>> Here's what I just checked in.  Can you check and see if it works for you?
> 
> No, It didn't worked for me.  I did a small change to it and now it works.

Thanks, I've checked in your fix to my fix.  I don't know how I messed
up your original change.

If you could, see if what's checked in works for you now.

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)

Re: More user-space probes

[Srinivasa D S]

On Friday 25 April 2008 07:55:41 pm David Smith wrote:
> Srinivasa D S wrote:
> > On Thursday 24 April 2008 11:25:40 pm David Smith wrote:
> >> Srinivasa D S wrote:
> >>> On Tuesday 22 April 2008 07:10:32 pm David Smith wrote:
> >>
> >> Here's what I just checked in.  Can you check and see if it works for
> >> you?
> >
> > No, It didn't worked for me.  I did a small change to it and now it
> > works.
>
> Thanks, I've checked in your fix to my fix.  I don't know how I messed
> up your original change.
>
> If you could, see if what's checked in works for you now.

It works well. 

Thanks
 Srinivasa DS

Re: More user-space probes

[Srinivasa D S]

On Friday 25 April 2008 07:55:41 pm David Smith wrote:
> Srinivasa D S wrote:
> > On Thursday 24 April 2008 11:25:40 pm David Smith wrote:
> >> Srinivasa D S wrote:
> >>> On Tuesday 22 April 2008 07:10:32 pm David Smith wrote:
> >>
> >> Here's what I just checked in.  Can you check and see if it works for
> >> you?
> >
> > No, It didn't worked for me.  I did a small change to it and now it
> > works.
>
> Thanks, I've checked in your fix to my fix.  I don't know how I messed
> up your original change.
>
> If you could, see if what's checked in works for you now.

It works well. 

Thanks
 Srinivasa DS

Re: More user-space probes

[Ananth N Mavinakayanahalli]

On Tue, Apr 22, 2008 at 08:40:32AM -0500, David Smith wrote:
> I've just finished implementing some new forms of user-space probing.
> Here's an example script:
> 
>   probe process("/bin/ls").exec { print("ls exec'ed!\n") }
>   probe process("/bin/ls").syscall { printf("|%d", $syscall) }
>   probe process("/bin/ls").syscall.return { print("+") }
>   probe process("/bin/ls").death { print("\nls done!\n") }
> 
> There are 5 new probe variants:
> 
>   process(PID_OR_PATH).clone
>   process(PID_OR_PATH).exec
>   process(PID_OR_PATH).death
>   process(PID_OR_PATH).syscall
>   process(PID_OR_PATH).syscall.return
> 
> (Internally, these are implemented using utrace.  These probe types also
> use the new "task_finder" framework, which provides the support by
> probing by PID or by PATH.  Eventually, the existing uprobes probes (and
> the future itrace probes) should be converted to use the "task_finder"
> framework.)

Dave,

Would it be possible to update the language reference with the new
constructs? I guess the langref needs an update for the marker support
too.

Ananth

Re: More user-space probes

[Ananth N Mavinakayanahalli]

On Tue, Apr 22, 2008 at 08:40:32AM -0500, David Smith wrote:
> I've just finished implementing some new forms of user-space probing.
> Here's an example script:
> 
>   probe process("/bin/ls").exec { print("ls exec'ed!\n") }
>   probe process("/bin/ls").syscall { printf("|%d", $syscall) }
>   probe process("/bin/ls").syscall.return { print("+") }
>   probe process("/bin/ls").death { print("\nls done!\n") }
> 
> There are 5 new probe variants:
> 
>   process(PID_OR_PATH).clone
>   process(PID_OR_PATH).exec
>   process(PID_OR_PATH).death
>   process(PID_OR_PATH).syscall
>   process(PID_OR_PATH).syscall.return
> 
> (Internally, these are implemented using utrace.  These probe types also
> use the new "task_finder" framework, which provides the support by
> probing by PID or by PATH.  Eventually, the existing uprobes probes (and
> the future itrace probes) should be converted to use the "task_finder"
> framework.)
> 
> Feel free to try out the new probe types and let me know if you find any
> issues.

Dave,

I am trying this on a multi-threaded process, I just get the traces for
the tgid. Running..

probe process(<path to multithreaded proc>).syscall
{
        printf("%d thread[%d] syscall\n", pid(), tid())
}

on a process with 2 threads yields:
...
21850 thread[21850] syscall
21850 thread[21850] syscall
21850 thread[21850] syscall
21850 thread[21850] syscall
...

Where 21850 is the pid.

I suppose support for following thread creation is a TODO? Or am I
missing something?

Ananth