RE: Fwd: Any way to find the network usage by a process?

RE: Fwd: Any way to find the network usage by a process?

[Stone, Joshua I]

On Thursday, October 05, 2006 5:00 PM, Frank Ch. Eigler wrote:
> Hi -
> 
> On Thu, Oct 05, 2006 at 04:28:10PM -0700, Mike Mason wrote:
>> [...]
>> ERROR: empty aggregate near identifier 'execname' at nettop.stp:35:4
>> WARNING: Number of errors: 1, skipped probes: 0
>> Apparently using @sum on empty aggregates isn't allowed. I expected
>> 0's to be returned.
> 
> As a judgement call, to be consistent with other extractors like @avg,
> the @sum etc. of an empty set was deemed to be undefined.

FWIW, Python allows you to take the sum of an empty list, even though
other equivalent extractors fail:

>>> sum([])
0
>>> max([])
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ValueError: min() or max() arg is an empty sequence

I'm just looking for precedence in other languages...

>> The only way to avoid the error is use @sum only if @count >
>> 0, which makes the printf too complex in my opinion.
> 
> Maybe so.  It's worth considering some syntactic sugar to express a
> undefined=>0 intent.

One possibility is an optional second parameter to the extractor that
gives a default value.  We could require this to be a constant (usually
0), or maybe even allow a numeric expression.

	a <<< 1; del a // prepare an empty aggregate
	printf("%d\n", @sum(a)) // runtime error
	printf("%d\n", @sum(a, 0)) // prints 0
	printf("%d\n", @sum(a, x)) // prints value of x

This works for min, max, and avg as well.

Or, instead of a default, it might be less confusing to consider the
second parameter as an initializer -- so when 'a' isn't empty, @sum(a,x)
== x + @sum(a).  Then min and max would effectively get an upper and
lower bound, respectively.  I'm not sure if avg works well this way
though.


Josh

Re: Fwd: Any way to find the network usage by a process?

[Jose R. Santos]

Irfan Habib wrote:
> Hi,
>
> Is there any method either kernel or user level which tells me which
> process is generating how much traffic from a machine. For example if
> some process is flooding the network, then I would like to know which
> process (PID ideally), is generating the most traffic.
>
>   

A while ago I did a SystemTap script to solve a problem similar to 
this.  It's been siting in my system for a while collecting dust and you 
currently don't need the embedded C code since the networking.stp tapset 
has all this script needs(and more), but I should point you in the right 
direction.

This worked a couple of months ago but it is currently untested.  Hope 
it helps.

-JRS


global ifstats, ifdevs, execname

%{
#include<linux/skbuff.h>
#include<linux/netdevice.h>
%}

probe kernel.function("dev_queue_xmit")
{
        execname[pid()] = execname()
        name=skb_to_name($skb)
        ifdevs[name] = name
        ifstats[pid(),name] <<< 1
}

function skb_to_name:string (skbuff:long)
%{
        struct sk_buff *skbuff = (struct sk_buff *)((long)THIS->skbuff);
        struct net_device *netdev = skbuff->dev;
        sprintf (THIS->__retvalue, "%s" , netdev->name);
%}

probe timer.ms(5000)
{
        exit()
}

probe end {
        foreach( pid in execname) {
                if (pid == 0) continue
                printf("%15s[%5d] ->\t", execname[pid],pid)
                foreach( ifname in ifdevs) {
                        printf("[%s:%7d] \t", ifname, 
@count(ifstats[pid, ifname]))
                }
                print("\n")
        }
        print("\n")
}

Re: Fwd: Any way to find the network usage by a process?

[Mike Mason]

Here's a variation of Jose's script that uses the networking tapset and 
prints top-like output for transmits and receives.  Much of the activity 
shows up under pid 0, which Jose's script filtered out.  This obviously 
doesn't reflect the actual process generating the traffic.

The networking tapset currently probes netif_receive_skb() for receives and 
dev_queue_xmit() for transmits.  Can anyone suggest better probe points to 
get transmits and receives by pid?

- Mike

Sample output:

   PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
     0     0 eth0        493     880      32    1238 swapper
  3078     0 eth0         26       2      28       2 Xvnc
13957     0 eth0          1       2       0       2 lspci
  3148     0 eth0          1       2       0       2 nautilus
  5058     0 eth0          1       1       0       1 firefox-bin
12277     0 eth0          1       0       0       0 sshd

nettop.stp script:

global ifxmit_p, ifrecv_p, ifxmit_b, ifrecv_b, ifdevs, ifpid, execname, user

probe netdev.transmit
{
         execname[pid()] = execname()
         user[pid()] = uid()
         ifdevs[pid(), dev_name] = dev_name
         ifxmit_p[pid(), dev_name] ++
         ifxmit_b[pid(), dev_name] += length
         ifpid[pid(), dev_name] ++
}

probe netdev.receive
{
         execname[pid()] = execname()
         user[pid()] = uid()
         ifdevs[pid(), dev_name] = dev_name
         ifrecv_p[pid(), dev_name] ++
         ifrecv_b[pid(), dev_name] += length
         ifpid[pid(), dev_name] ++
}


function print_activity()
{
         printf("%5s %5s %-7s %7s %7s %7s %7s %-15s\n",
                 "PID", "UID", "DEV", "XMIT_PK", "RECV_PK", "XMIT_KB",
		"RECV_KB", "COMMAND")

         foreach ([pid,dev] in ifpid-) {
                 printf("%5d %5d %-7s %7d %7d %7d %7d %-15s\n",
                         pid, user[pid], dev,
			ifxmit_p[pid, dev], ifrecv_p[pid, dev],                        		 
ifxmit_b[pid, dev]/1024,
  			ifrecv_b[pid, dev]/1024,
                         execname[pid])
         }

         print("\n")

         delete execname
         delete user
         delete ifdevs
         delete ifxmit_p
         delete ifrecv_p
         delete ifxmit_b
         delete ifrecv_b
         delete ifpid
}

probe timer.ms(5000)
{
         print_activity()
}


Jose R. Santos wrote:
> Irfan Habib wrote:
>> Hi,
>>
>> Is there any method either kernel or user level which tells me which
>> process is generating how much traffic from a machine. For example if
>> some process is flooding the network, then I would like to know which
>> process (PID ideally), is generating the most traffic.
>>
>>   
> 
> A while ago I did a SystemTap script to solve a problem similar to 
> this.  It's been siting in my system for a while collecting dust and you 
> currently don't need the embedded C code since the networking.stp tapset 
> has all this script needs(and more), but I should point you in the right 
> direction.
> 
> This worked a couple of months ago but it is currently untested.  Hope 
> it helps.
> 
> -JRS
> 
> 
> global ifstats, ifdevs, execname
> 
> %{
> #include<linux/skbuff.h>
> #include<linux/netdevice.h>
> %}
> 
> probe kernel.function("dev_queue_xmit")
> {
>        execname[pid()] = execname()
>        name=skb_to_name($skb)
>        ifdevs[name] = name
>        ifstats[pid(),name] <<< 1
> }
> 
> function skb_to_name:string (skbuff:long)
> %{
>        struct sk_buff *skbuff = (struct sk_buff *)((long)THIS->skbuff);
>        struct net_device *netdev = skbuff->dev;
>        sprintf (THIS->__retvalue, "%s" , netdev->name);
> %}
> 
> probe timer.ms(5000)
> {
>        exit()
> }
> 
> probe end {
>        foreach( pid in execname) {
>                if (pid == 0) continue
>                printf("%15s[%5d] ->\t", execname[pid],pid)
>                foreach( ifname in ifdevs) {
>                        printf("[%s:%7d] \t", ifname, @count(ifstats[pid, 
> ifname]))
>                }
>                print("\n")
>        }
>        print("\n")
> }
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Fwd: Any way to find the network usage by a process?

[Frank Ch. Eigler]

Mike Mason <mmlnx@us.ibm.com> writes:

> Here's a variation of Jose's script that uses the networking tapset
> and prints top-like output for transmits and receives.  [...]

Thanks for posting it to the systemtap wiki.

Some minor style suggestions follow:

> [...]
>          ifxmit_p[pid(), dev_name] ++
>          ifxmit_b[pid(), dev_name] += length

These could be collapsed into a single statistics-aggregate array: 
#          ifxmit[pid(), dev_name] <<< length
Then the printing routine would use @count(ifxmit[...]) and @sum(ifxmit[...])
to extract the two values.  Same of course for ifrecv.

>          execname[pid()] = execname()
>          user[pid()] = uid()
>          ifdevs[pid(), dev_name] = dev_name

Calling pid() so many times is worse than calling it once and caching
the result in a local variable ("p = pid()").  

The way that the script tracks pid-to-uid and pid-to-execname mappings
is not bad, though if that part were moved to new probes on fork or
exec, it would allow the network-related probes to run concurrently on
an SMP without fighting over locks.


- FChE

Re: Fwd: Any way to find the network usage by a process?

[Mike Mason]

Frank Ch. Eigler wrote:
> Mike Mason <mmlnx@us.ibm.com> writes:
> 
>> Here's a variation of Jose's script that uses the networking tapset
>> and prints top-like output for transmits and receives.  [...]
> 
> Thanks for posting it to the systemtap wiki.
> 
> Some minor style suggestions follow:
> 
>> [...]
>>          ifxmit_p[pid(), dev_name] ++
>>          ifxmit_b[pid(), dev_name] += length
> 
> These could be collapsed into a single statistics-aggregate array: 
> #          ifxmit[pid(), dev_name] <<< length
> Then the printing routine would use @count(ifxmit[...]) and @sum(ifxmit[...])
> to extract the two values.  Same of course for ifrecv.

I tried that and got the following output:

   PID   UID DEV     XMIT_PK RECV_PK XMIT_KB RECV_KB COMMAND
     0     0 eth0          9      10     486     672 swapper
ERROR: empty aggregate near identifier 'execname' at nettop.stp:35:4
WARNING: Number of errors: 1, skipped probes: 0

Apparently using @sum on empty aggregates isn't allowed. I expected 0's to 
be returned. The only way to avoid the error is use @sum only if @count > 
0, which makes the printf too complex in my opinion.

> 
>>          execname[pid()] = execname()
>>          user[pid()] = uid()
>>          ifdevs[pid(), dev_name] = dev_name
> 
> Calling pid() so many times is worse than calling it once and caching
> the result in a local variable ("p = pid()").  

Agreed.  I'll change that.

> 
> The way that the script tracks pid-to-uid and pid-to-execname mappings
> is not bad, though if that part were moved to new probes on fork or
> exec, it would allow the network-related probes to run concurrently on
> an SMP without fighting over locks.

But that would only catch processes created after the script starts, correct?

- Mike

> 
> 
> - FChE

Re: Fwd: Any way to find the network usage by a process?

[Frank Ch. Eigler]

Hi -

On Thu, Oct 05, 2006 at 04:28:10PM -0700, Mike Mason wrote:
> [...]
> ERROR: empty aggregate near identifier 'execname' at nettop.stp:35:4
> WARNING: Number of errors: 1, skipped probes: 0
> Apparently using @sum on empty aggregates isn't allowed. I expected 0's to 
> be returned. 

As a judgement call, to be consistent with other extractors like @avg,
the @sum etc. of an empty set was deemed to be undefined.

> The only way to avoid the error is use @sum only if @count > 
> 0, which makes the printf too complex in my opinion.

Maybe so.  It's worth considering some syntactic sugar to express a
undefined=>0 intent.

> >The way that the script tracks pid-to-uid and pid-to-execname mappings
> >is not bad, though if that part were moved to new probes on fork or
> >exec, it would allow the network-related probes to run concurrently on
> >an SMP without fighting over locks.
> 
> But that would only catch processes created after the script starts, 
> correct?

That's true.

- FChE

Re: Fwd: Any way to find the network usage by a process?

[Mike Mason]

Frank Ch. Eigler wrote:
> Hi -
> 
> On Thu, Oct 05, 2006 at 04:28:10PM -0700, Mike Mason wrote:
>> [...]
>> ERROR: empty aggregate near identifier 'execname' at nettop.stp:35:4
>> WARNING: Number of errors: 1, skipped probes: 0
>> Apparently using @sum on empty aggregates isn't allowed. I expected 0's to 
>> be returned. 
> 
> As a judgement call, to be consistent with other extractors like @avg,
> the @sum etc. of an empty set was deemed to be undefined.
> 
>> The only way to avoid the error is use @sum only if @count > 
>> 0, which makes the printf too complex in my opinion.
> 
> Maybe so.  It's worth considering some syntactic sugar to express a
> undefined=>0 intent.

Actually, the following works fine and doesn't complicate it too much:

         foreach ([pid, dev] in ifpid-) {
                 n_xmit = @count(ifxmit[pid, dev])
                 n_recv = @count(ifrecv[pid, dev])
                 printf("%5d %5d %-7s %7d %7d %7d %7d %-15s\n",
                         pid, user[pid], dev, n_xmit, n_recv,
                         n_xmit ? @sum(ifxmit[pid, dev])/1024 : 0,
                         n_recv ? @sum(ifrecv[pid, dev])/1024 : 0,
                         execname[pid])
         }

I'll update the wiki page.  Thanks for the suggestions.

- Mike

Re: Fwd: Any way to find the network usage by a process?

[Frank Ch. Eigler]

Mike Mason <mmlnx@us.ibm.com> writes:

> [...]
> >> The only way to avoid the error is use @sum only if @count > 0,
> >> which makes the printf too complex in my opinion.
> [...]

Actually, another way is to add a "<<< 0" dummy value to the
stats arrays in a new "probe begin", then to display @count()-1
and @sum() etc.

- FChE