Tracking vm activity

Tracking vm activity

[William Cohen]

[-- Attachment #1: Type: text/plain, Size: 941 bytes --]

Hi,

I am playing around with getting information about page faults and I noticed 
that there is a probe alias for the entry for the pagefault code. However, there 
is no matching probe point for the return. It is useful to look at the return 
value to determine what kind of page fault occurred (major or minor). The 
attached patch provides a similar probe point for the return point. any comments 
on the patch?

This would be useful for the following senario. A probe on vm.pagefault could 
get the address and a probe on vm.pagefault.return could get information how it 
was handled. The analysis could then do things like which addresses cause major 
page faults (real disk accesses). One could write out a log of the major page 
faults (and the mmap operations) and track which files caused the page faults. 
Probably don't want to walk through the mm structures when taking a page fault 
to determine which file it came from.

-Will

[-- Attachment #2: pagefault_return.patch --]
[-- Type: text/x-patch, Size: 595 bytes --]

Index: tapset/memory.stp
===================================================================
RCS file: /cvs/systemtap/src/tapset/memory.stp,v
retrieving revision 1.4
diff -U2 -u -r1.4 memory.stp
--- tapset/memory.stp	7 Nov 2006 09:26:24 -0000	1.4
+++ tapset/memory.stp	21 Mar 2007 14:40:27 -0000
@@ -27,4 +27,10 @@
 }
 
+probe vm.pagefault.return = kernel.function(
+        %( kernel_v >= "2.6.13" %? "__handle_mm_fault" %: "handle_mm_fault" %)
+        ).return
+{
+}
+
 /* Return which node the given address belongs to in a NUMA system */
 function addr_to_node:long(addr:long)  /* pure */ 

Re: Tracking vm activity

[Frank Ch. Eigler]

William Cohen <wcohen@redhat.com> writes:

> I am playing around with getting information about page faults and I
> noticed that there is a probe alias for the entry for the pagefault
> code. However, there is no matching probe point for the
> return. [...]

Sure.  Please also make sure that some test case attempts to
instantiate this new probe, and that it is at least as well documented
as its neighbours.

- FChE

Re: Tracking vm activity

[Stone, Joshua I]

William Cohen wrote:
> I am playing around with getting information about page faults and I 
> noticed that there is a probe alias for the entry for the pagefault 
> code. However, there is no matching probe point for the return. It is 
> useful to look at the return value to determine what kind of page fault 
> occurred (major or minor). The attached patch provides a similar probe 
> point for the return point. any comments on the patch?

I see that you committed it, and added a variable for the return value 
(fault_type).  Could you elaborate in the documentation what the 
possible types are (major, minor) and their values?

Thanks,

Josh

Re: Tracking vm activity

[William Cohen]

Stone, Joshua I wrote:
> William Cohen wrote:
>> I am playing around with getting information about page faults and I 
>> noticed that there is a probe alias for the entry for the pagefault 
>> code. However, there is no matching probe point for the return. It is 
>> useful to look at the return value to determine what kind of page 
>> fault occurred (major or minor). The attached patch provides a similar 
>> probe point for the return point. any comments on the patch?
> 
> I see that you committed it, and added a variable for the return value 
> (fault_type).  Could you elaborate in the documentation what the 
> possible types are (major, minor) and their values?
> 
> Thanks,
> 
> Josh

Hi Josh,

Is this suitable for comment for vm.pagefault.return? If suitable, I will go 
ahead and check it in.


/* probe vm.pagefault.return
  *
  *  Records type of fault that occurred.
  *
  * Context:
  *  The process which triggered the fault.
  *
  * Argumentss:
  *  fault_type - type of fault
  *	VM_FAULT_OOM	0	out of memory
  *	VM_FAULT_SIGBUS	1	if not oom, minor, or major fault, this val
  *	VM_FAULT_MINOR	2	no blocking operation to handle fault
  *	VM_FAULT_MAJOR	3	required blocking operation to handle fault
  */

-Will

Re: Tracking vm activity

[Stone, Joshua I]

William Cohen wrote:
> Is this suitable for comment for vm.pagefault.return? If suitable, I 
> will go ahead and check it in.
> [...]
>  *  fault_type - type of fault
>  *    VM_FAULT_OOM    0    out of memory
>  *    VM_FAULT_SIGBUS    1    if not oom, minor, or major fault, this val
>  *    VM_FAULT_MINOR    2    no blocking operation to handle fault
>  *    VM_FAULT_MAJOR    3    required blocking operation to handle fault

Yes, that's exactly what I was looking for, thanks.

Josh

Re: Tracking vm activity

[Jun Koi]

On 4/24/07, William Cohen <wcohen@redhat.com> wrote:
> Hi,
>
> I am playing around with getting information about page faults and I noticed
> that there is a probe alias for the entry for the pagefault code. However, there
> is no matching probe point for the return. It is useful to look at the return
> value to determine what kind of page fault occurred (major or minor). The
> attached patch provides a similar probe point for the return point. any comments
> on the patch?
>
> This would be useful for the following senario. A probe on vm.pagefault could
> get the address and a probe on vm.pagefault.return could get information how it
> was handled. The analysis could then do things like which addresses cause major
> page faults (real disk accesses). One could write out a log of the major page
> faults (and the mmap operations) and track which files caused the page faults.
> Probably don't want to walk through the mm structures when taking a page fault
> to determine which file it came from.
>

Hi Will,

I found this idea interesting. However, what do you think if the
machine is SMP, or kernel is preemptive capable?

Just imagine a scenario like this with 2 kernel threads A and B:
- A causes page fault, then enters page fault handler (1)
- B preempts A, causes fault, then enters page fault handler  (2)
- A exits page fault handler (3)
- B exits page fault handler (4)

In this case, you get the exit value at (3), but it is not relevant to
the parameters collected in the above step (2).

I cannot figure out how to correlate "enter" and "exit" function
events in SMP or preemptive case.

Any idea?

Thanks,
Jun

Re: Tracking vm activity

[Frank Ch. Eigler]

"Jun Koi" <junkoi2004@gmail.com> writes:

> [...]
> I found this idea interesting. However, what do you think if the
> machine is SMP, or kernel is preemptive capable?
> [...]
> I cannot figure out how to correlate "enter" and "exit" function
> events in SMP or preemptive case.

In systemtap, a .return probe can refer to parameters of the
*corresponding* .call (function entry) event.  The parameters are
saved in an auxiliary lookup table, indexed by thread-id and nesting
level.  SMP or preemption should not cause any problem.

- FChE

Re: Tracking vm activity

[Jun Koi]

On 08 May 2007 10:15:43 -0400, Frank Ch. Eigler <fche@redhat.com> wrote:
> "Jun Koi" <junkoi2004@gmail.com> writes:
>
> > [...]
> > I found this idea interesting. However, what do you think if the
> > machine is SMP, or kernel is preemptive capable?
> > [...]
> > I cannot figure out how to correlate "enter" and "exit" function
> > events in SMP or preemptive case.
>
> In systemtap, a .return probe can refer to parameters of the
> *corresponding* .call (function entry) event.  The parameters are
> saved in an auxiliary lookup table, indexed by thread-id and nesting
> level.  SMP or preemption should not cause any problem.

Ah, the thread-id idea is clear to me. But may you explain a little
bit about the "nesting level"??

Thanks you,
Jun

Re: Tracking vm activity

[Frank Ch. Eigler]

Hi -

On Thu, May 10, 2007 at 07:38:04PM +0900, Jun Koi wrote:
> Ah, the thread-id idea is clear to me. But may you explain a little
> bit about the "nesting level"??

Certainly.  The nesting level is a per-thread counter.  It is
incremented on a probed function entry and decremented upon its
return.  It is used as an additional index to saved function
parameters by.  This way, if a recursive function is encountered, we
can find the innermost parameters without losing the outer ones.

- FChE

Re: Tracking vm activity

[Jun Koi]

Hi Frank,

> > Ah, the thread-id idea is clear to me. But may you explain a little
> > bit about the "nesting level"??
>
> Certainly.  The nesting level is a per-thread counter.  It is
> incremented on a probed function entry and decremented upon its
> return.  It is used as an additional index to saved function
> parameters by.  This way, if a recursive function is encountered, we
> can find the innermost parameters without losing the outer ones.
>

It is all clear now. Thanks, Frank :-)

Best,
Jun

Re: Tracking vm activity

[William Cohen]

Jun Koi wrote:
> On 4/24/07, William Cohen <wcohen@redhat.com> wrote:
>> Hi,
>>
>> I am playing around with getting information about page faults and I 
>> noticed
>> that there is a probe alias for the entry for the pagefault code. 
>> However, there
>> is no matching probe point for the return. It is useful to look at the 
>> return
>> value to determine what kind of page fault occurred (major or minor). The
>> attached patch provides a similar probe point for the return point. 
>> any comments
>> on the patch?
>>
>> This would be useful for the following senario. A probe on 
>> vm.pagefault could
>> get the address and a probe on vm.pagefault.return could get 
>> information how it
>> was handled. The analysis could then do things like which addresses 
>> cause major
>> page faults (real disk accesses). One could write out a log of the 
>> major page
>> faults (and the mmap operations) and track which files caused the page 
>> faults.
>> Probably don't want to walk through the mm structures when taking a 
>> page fault
>> to determine which file it came from.
>>
> 
> Hi Will,
> 
> I found this idea interesting. However, what do you think if the
> machine is SMP, or kernel is preemptive capable?
> 
> Just imagine a scenario like this with 2 kernel threads A and B:
> - A causes page fault, then enters page fault handler (1)
> - B preempts A, causes fault, then enters page fault handler  (2)
> - A exits page fault handler (3)
> - B exits page fault handler (4)
> 
> In this case, you get the exit value at (3), but it is not relevant to
> the parameters collected in the above step (2).
> 
> I cannot figure out how to correlate "enter" and "exit" function
> events in SMP or preemptive case.
> 
> Any idea?
> 
> Thanks,
> Jun

It is quite possible that more than one application has outstanding page faults. 
By definition a context switch happens when a major page fault occurs. However, 
associative arrays can be used to keep that kind of matching information 
straight. Thus, this will not be a problem.

-Will