Re: PLEASE HELP regarding System Tap

Re: PLEASE HELP regarding System Tap

[Khushboo Goel]

>
> > I am using System Tap for probing kernel function so that I can know
> > more about packet that are send over network. For this I need to
> > open a File so that I can save all raw data. I want to save this
> > data when I probing in the function, continuously.
>
> We don't have a C stdio-like facility in systemtap yet.  You can make
> your systemtap probe handler just printf(...), which will come out at
> the stap(run) command's stdout.  You can redirect that as you like
> (perhaps using the "-o" option).  Is that sufficient?
>  ------

     When i use -o option, it gives me buffer overflow error, i am
anyhow not printing anything on screen, i am using an aggregate
variable to store all information, but i have huge data around 50K
packets in one run ,which gives me overflow buffer , i also increased
the stack size but if i increase the number of packets i send over the
network, it gives me same error.
>
>
> > I using FILE *fp = NULL, to make file pointer, but its not allowing
> > to me do it.
>
> Right, that would be C code that's appropriate to run in a user-space
> program.  Systemtap probe scripts resemble C, but not that much.  Plus
> as any kernel-side code, they cannot just invoke user-space libraries
> like stdio.
>
>
> - FChE

Re: PLEASE HELP regarding System Tap

[Frank Ch. Eigler]

Hi -
On Mon, Jul 07, 2008 at 01:30:55PM -0700, Khushboo Goel wrote:
> [...]
>      When i use -o option, it gives me buffer overflow error, i am
> anyhow not printing anything on screen, 

OK, then "-o" won't matter.

> i am using an aggregate variable to store all information, but i
> have huge data around 50K packets in one run ,which gives me
> overflow buffer[...]

You may need to use a larger MAXMAPENTRIES (see the stap(1) man page),
or explicitly size the buffering aggregate global

   global buffer[50000]

I wonder if your array indexing makes sense though.  Can you post your
full script?


- FChE

Re: PLEASE HELP regarding System Tap

[Masami Hiramatsu]

Hi,

Khushboo Goel wrote:
>>> I am using System Tap for probing kernel function so that I can know
>>> more about packet that are send over network. For this I need to
>>> open a File so that I can save all raw data. I want to save this
>>> data when I probing in the function, continuously.
>> We don't have a C stdio-like facility in systemtap yet.  You can make
>> your systemtap probe handler just printf(...), which will come out at
>> the stap(run) command's stdout.  You can redirect that as you like
>> (perhaps using the "-o" option).  Is that sufficient?
>>  ------
> 
>      When i use -o option, it gives me buffer overflow error, i am
> anyhow not printing anything on screen, i am using an aggregate
> variable to store all information, but i have huge data around 50K
> packets in one run ,which gives me overflow buffer , i also increased
> the stack size but if i increase the number of packets i send over the
> network, it gives me same error.

Could you try to expand your buffer and use bulk transfer mode?
you can specify buffer size for each cpu in MB by -s option(sysemtap
uses only 256KB single buffer without -s option, it's too small
for you), and use bulk transfer mode by -b option.
When you uses bulk mode, you can see raw output data in
stpd_cpu* files. these data are output per cpu, so I think
it's good for your situation.

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America) Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: PLEASE HELP regarding System Tap

[Khushboo Goel]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

On Mon, Jul 7, 2008 at 1:45 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> Hi -
> On Mon, Jul 07, 2008 at 01:30:55PM -0700, Khushboo Goel wrote:
>> [...]
>>      When i use -o option, it gives me buffer overflow error, i am
>> anyhow not printing anything on screen,
>
> OK, then "-o" won't matter.
>
>> i am using an aggregate variable to store all information, but i
>> have huge data around 50K packets in one run ,which gives me
>> overflow buffer[...]
>
> You may need to use a larger MAXMAPENTRIES (see the stap(1) man page),
> or explicitly size the buffering aggregate global
>
>   global buffer[50000]
>

I don't want to use variable like this as, it doesn't solve my problem.

> I wonder if your array indexing makes sense though.  Can you post your
> full script?
>
Yea i am posting my full script for now, its under work....  I have
commented most of thing.


Is it possible in System Tap that i can find address of the location
in the memory which is assign to the buffer?? So that i can write
directly to the location.
>
> - FChE
>

[-- Attachment #2: tcp_check_FILE.stap~ --]
[-- Type: application/octet-stream, Size: 6975 bytes --]

//-----------------------------------------------------------------------------------------------------//
//Author        :       Khushboo Goel
//Description   :       Goal is to profile the tcp_rcv_established() function in the kernel
//
//Output        :       1. Average packet size on a given connection
//                      2. histogram of packet size on any connection
//Revisions     :
//      		Trying to open FILE using Embedded C
//
//-----------------------------------------------------------------------------------------------------//

global npr_totpktsize, npr_pktcount
global npr_pureack_count, npr_datapkt_count
global npr_connid
global npr_pktid
global npr_init_time

//statistics variable by connectionid
global npr_pktsize
global npr_pkttime

%{
   #include <linux/version.h>
   #include <net/sock.h>
   #include <net/tcp.h>
   #include <net/ip.h>
   #include <linux/skbuff.h>
%}

probe begin
{
//      printf("Starting probe....\n");

        //initialize all global variables
        npr_totpktsize = 0
        npr_pktcount = 0
        npr_pureack_count = 0
        npr_datapkt_count = 0
        npr_connid=""
        npr_pktid=0
//      npr_init_time = gettimeofday_ns();
//      printf("timestamp: %d(%x)",npr_init_time,npr_init_time);
}


//npr
probe kernel.function("tcp_rcv_established")
{
        npr_sourceport  = ntohs($th->source);
        npr_destport    = ntohs($th->dest);
        npr_datalen     = $skb->len; //npr added only for debug print only

//DEBUG
//if(1)
//      printf("\n%s : len=%d th.source =%d dest=%d datalen=%d", probefunc(), $len, npr_sourceport, npr_destport,npr_datalen);
//DEBUG

   	  npr_sourceipaddr = npr_get_source_ipaddr($skb);
     //	  npr_destipaddr = npr_get_dest_ipaddr($skb);
     	  npr_tcphdr_len = npr_tcphdr_len($sk);
//DEBUG
//if(1)
 //     printf("\n\n sourceip= %s   destip= %s  tcpheaderlength:%d",npr_sourceipaddr,npr_destipaddr, npr_tcphdr_len);
//DEBUG

        //connectionid string (source ipaddr,sourceport, destipaddr, destport)
        npr_connid=npr_sourceipaddr."_".sprint(npr_sourceport)."_".npr_destipaddr."_".sprint(npr_destport)
        //printf(npr_connid, "%s_%d_%s_%d", npr_sourceipaddr,npr_sourceport,npr_destipaddr,npr_destport);

//DEBUG
//if(1)
//      printf("\n ConnectionID : %s\n", npr_connid);
//DEBUG

        //update the pktsize aggregate
        npr_pktsize[npr_connid] <<< $len;
        npr_totpktsize += $len
        npr_pktcount++
//----------------------------------WIP----------------------------
        npr_pktid++
        //stat object to hold timestamp of receive packets packets
//      npr_curpkt_time = gettimeofday_ns() - npr_init_time;
//      printf("\tcur pkt time:%d\n",npr_curpkt_time);
  //    npr_pkttime[npr_connid] <<< npr_curpkt_time;
//-------------------------------------------------------------------
        if($len > npr_tcphdr_len) {
                npr_datapkt_count++
        //      printf("\ndata packet");
        }
        else {
        //      printf("\n ACK PACKET:%d", npr_pureack_count);
                npr_pureack_count++
        }
}

probe end
{
   foreach([npr_connid] in npr_pktsize)
 {
    printf("\n\t Tcp_rcv_established() - Packet SIZE Statistics");
    printf("\n\t~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");
    printf("\n ConnectionID: %s", npr_connid);
    printf("\n Total Pkts Received :%d  Total Pkt Size: %d   Avg Pkt Size = %d",npr_pktcount, npr_totpktsize, npr_totpktsize/npr_pktcount);
    printf("\n PureAck Pkt Count : %d   Payload Pkts count:%d\n\n", npr_pureack_count,npr_datapkt_count);
    print(@hist_linear(npr_pktsize[npr_connid],1,5000,50));
    printf("\n=======================================================\n");
}

/*    printf("\n\t Tcp_rcv_established() - Packet timestamp Statistics");
    printf("\n\t~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~");
    printf("\n Total Pkts Received :%d  Total Pkt Size: %d   Avg Pkt Size = %d",npr_pktcount, npr_totpktsize, npr_totpktsize/npr_pktcount);
    printf("\n PureAck Pkt Count : %d   Payload Pkts count:%d\n\n", npr_pureack_count,npr_datapkt_count);
    print(@hist_linear(npr_pkttime[npr_connid],1,100000,30000));
    printf("\n=======================================================\n");
*/
}

//
//print packet statistics every 4 seconds
/*
probe timer.ms(4000)
{
   //printf("%11d %10d %10d %10d %10d\n", udp_out, udp_outerr, udp_in, udp_inerr, udp_noport)
   //npr22 add code here to display status intermediately

}

*/

//NPR+
// all utility functions listed below
//
//Function to get TCP header length

function npr_tcphdr_len:long(npr_sk:long)
%{
    struct tcp_sock *npr_tcpsock = (struct tcp_sock*)(long)THIS->npr_sk;
    THIS->__retvalue = npr_tcpsock->tcp_header_len;
%}

//

//function to extract and return the source ipaddress
//from ip header

function npr_get_source_ipaddr:string(npr_skb:long)
%{
        //THIS->__retvalue = 123;
        //unsigned long nprskbsource;
        //char *npr_saddrptr;
        //unsigned long npr_saddr;
        struct sk_buff  *npr_curskb;
        struct iphdr *npr_iphdr;
        npr_curskb = (struct sk_buff *)(long)THIS->npr_skb;
        //calling ip layer functions
        npr_iphdr = ip_hdr(npr_curskb);
        sprintf(THIS->__retvalue,"%x",ntohl(npr_iphdr->saddr));
      // npr_saddr = ntohl(npr_iphdr->saddr);
      // npr_saddrptr = (char*)&npr_saddr;
      // sprintf(THIS->__retvalue,"%u.%u.%u.%u",*(npr_saddrptr+3), *(npr_saddrptr+2),*(npr_saddrptr+1),*(npr_saddrptr));
%}

//function to extract and return the dest ipaddress
//from ip header
/*
function npr_get_dest_ipaddr:string(npr_skb:long)
%{
        struct sk_buff  *npr_curskb;
        struct iphdr *npr_iphdr;
        npr_curskb = (struct sk_buff *)(long)THIS->npr_skb;

        npr_iphdr = ip_hdr(npr_curskb);
        sprintf(THIS->__retvalue,"%x",ntohl(npr_iphdr->daddr));
%}
*/
/************* Below are some sample instrumentation code i tried to explore - raja ************************
//      codes below not tested
//
probe kernel.function("tcp_v4_rcv")
{
 // printf("\ntcp_v4_rcv probe called");
}

probe kernel.function("icmp_send")
{
 // icmp_send(skb, icmp_dest_unreach, icmp_port_unreach,0);
  if($type == 3 && $code == 3)
        {
           //  if($skb_in->nh->iph->protocol == 17) udp_noport++
        }mydatalen
  printf("\n pkt_type=%d", $skb_in->pkt_type);
}
//

//probe kernel.function("ip_rcv")
probe kernel.function("tcp_v4_rcv")
{
        //printf("\npkt type=%d len=%d dev.irq=%d", $skb->pkt_type, $skb->len, $dev->irq);
   printf("\nskb->len=%d th->seqno:%d", $skb->len,  $th->seq);
   //mytcpheader = $skb->head+$skb->transport_header;
   //printf("\ntcp header dump: %s", $skb->head+$skb->transport_header);
}

probe kernel.function("kmalloc")
{
 //  printf("name=%s", probefunc());
}
probe timer.ms(1000)
{
   //printf("%11d %10d %10d %10d %10d\n", udp_out, udp_outerr, udp_in, udp_inerr, udp_noport)
}
****************************************************************/

Re: PLEASE HELP regarding System Tap

[Frank Ch. Eigler]

"Khushboo Goel" <kgoel@usc.edu> writes:

> [...]
>>> i am using an aggregate variable to store all information, but i
>>> have huge data around 50K packets in one run ,which gives me
>>> overflow buffer[...]
>>
>> You may need to use a larger MAXMAPENTRIES (see the stap(1) man page),
>> or explicitly size the buffering aggregate global
>>
>>   global buffer[50000]
>
> I don't want to use variable like this as, it doesn't solve my problem.

Could you explain why you think so?  You haven't transcribed your
error message(s) yet, so we are still only guessing.


> Yea i am posting my full script for now, its under work....  I have
> commented most of thing.

Thanks.

> Is it possible in System Tap that i can find address of the location
> in the memory which is assign to the buffer?? So that i can write
> directly to the location.

I'm sorry, I don't understand.  You should not have to use any unusual
measures just to produce tracing output.


Looking at your script, you are doing one space-hungry and one
time-hungry thing that may each need change.  Your npr_pktsize[] array
will be both tall and wide -- potentially many elements, each a wide
histogram.  That will consume lots of memory to store.  The whole
array will also take lots of time (statements) and (buffer) space to
print out with that unlimited foreach().  Depending on which resource
systemtap detects as exhausted, you may get different error messages.
For each of them, there is some sort of countermeasure.

I'll write up a wiki page with tips about the various alternatives,
but you could in the mean time share your errors so we can point you
to the specific one you're hitting.

- FChE

RE: PLEASE HELP regarding System Tap

[Stone, Joshua I]

Frank Ch. Eigler wrote:
> Looking at your script, you are doing one space-hungry and one
> time-hungry thing that may each need change.  Your npr_pktsize[] array
> will be both tall and wide -- potentially many elements, each a wide
> histogram.  That will consume lots of memory to store.  The whole
> array will also take lots of time (statements) and (buffer) space to
> print out with that unlimited foreach().  Depending on which resource
> systemtap detects as exhausted, you may get different error messages.
> For each of them, there is some sort of countermeasure.

Also, it might be helpful to find a reasonable "end-of-connection" probe point for a given connection id.  At that point you could print the accumulated data for that connection and delete it from the array.  If npr_pktsize only contains the currently active connections, you'll probably have a much better bound on the size.

Josh

Re: PLEASE HELP regarding System Tap

[Khushboo Goel]

I am working on what you have suggested, i will put a check when the
connection is over and will put delete that specific connection id
array.
Since its both space and time hungry so i have thought of alternative
way to write to life as
i need to save the data wen i am probing, so instead of putting in the
aggregate variable i am printing it to file, and checking how much its
effecting CPU Performance and instead of using histogram of the
systemTap , i will store it in some other file.

I had another question

Is it possible i can know what address location is assigned to system
Tap buffer in the memory ??


On Tue, Jul 8, 2008 at 1:45 PM, Stone, Joshua I
<joshua.i.stone@intel.com> wrote:
> Frank Ch. Eigler wrote:
>> Looking at your script, you are doing one space-hungry and one
>> time-hungry thing that may each need change.  Your npr_pktsize[] array
>> will be both tall and wide -- potentially many elements, each a wide
>> histogram.  That will consume lots of memory to store.  The whole
>> array will also take lots of time (statements) and (buffer) space to
>> print out with that unlimited foreach().  Depending on which resource
>> systemtap detects as exhausted, you may get different error messages.
>> For each of them, there is some sort of countermeasure.
>
> Also, it might be helpful to find a reasonable "end-of-connection" probe point for a given connection id.  At that point you could print the accumulated data for that connection and delete it from the array.  If npr_pktsize only contains the currently active connections, you'll probably have a much better bound on the size.
>
> Josh
>

Re: PLEASE HELP regarding System Tap

[Stone, Joshua I]

Khushboo Goel wrote:
> Is it possible i can know what address location is assigned to system
> Tap buffer in the memory ??

Well, yes, all of SystemTap is open source, so it's possible to peek
into the internals.  The runtime sources contain the implementation for
the maps data structures.  You can also run your script with "-p3" to
see the code that's generated, and use that as a guide to tell how the
maps are accessed.

It seems very strange that you would ask for this though.  What would
you do with the address once you find it?

Josh

Re: PLEASE HELP regarding System Tap

[Frank Ch. Eigler]

Hi -


> I am working on what you have suggested, i will put a check when the
> connection is over and will put delete that specific connection id
> array.

(Or even better - you could print the histogram for that connection at
that time.)


> Since its both space and time hungry so i have thought of
> alternative way to write to life as i need to save the data wen i am
> probing, so instead of putting in the aggregate variable i am
> printing it to file, and checking how much its effecting CPU
> Performance and instead of using histogram of the systemTap , i will
> store it in some other file.

You can do that too, but I wonder if your data needs are indeed as
large as that first systemtap script suggested.  It appeared to be
asking for a 100-bucket histogram (~1kB) for each of (say)
ten-thousand tcp connections (10 MB storage during run).  ASCII
pretty-printing each histogram could multiply the space by another
factor of ~5, resulting in about 50 MB of output.  Is this about
right?


> Is it possible i can know what address location is assigned to
> system Tap buffer in the memory ??

There are many different buffers.  You might be interested in the one
the runtime can hand over via _stp_reserve(), but using that is not
routinely recommended.

- FChE

Re: PLEASE HELP regarding System Tap

[Khushboo Goel]

>
>> I am working on what you have suggested, i will put a check when the
>> connection is over and will put delete that specific connection id
>> array.
>
> (Or even better - you could print the histogram for that connection at
> that time.)
>
>
>> Since its both space and time hungry so i have thought of
>> alternative way to write to life as i need to save the data wen i am
>> probing, so instead of putting in the aggregate variable i am
>> printing it to file, and checking how much its effecting CPU
>> Performance and instead of using histogram of the systemTap , i will
>> store it in some other file.
>
> You can do that too, but I wonder if your data needs are indeed as
> large as that first systemtap script suggested.  It appeared to be
> asking for a 100-bucket histogram (~1kB) for each of (say)
> ten-thousand tcp connections (10 MB storage during run).  ASCII
> pretty-printing each histogram could multiply the space by another
> factor of ~5, resulting in about 50 MB of output.  Is this about
> right?
>
>
 I am actually sending some 10Gb or data over 8 tcp connection from 1 cpu ..
But its post calculation so i don't think so i wana get into that ...


>> Is it possible i can know what address location is assigned to
>> system Tap buffer in the memory ??
>
> There are many different buffers.  You might be interested in the one
> the runtime can hand over via _stp_reserve(), but using that is not
> routinely recommended.

Can you tell me more about it or any documentation on it ??? i want to
go this runtime memory and write there directly.


----
I have observed one more thing , with the aggregate being in the code,
and i just remove one of the embedded C code , i don't get any buffer
overflow error, and it prints histogram and everything even for 20Gb
of data .. with NAGLE off and sending 64 Bytes packet ( 96Bytes
inclusive of header )
Which is strange coz i expected aggregate to show overflow buffer error
>
> - FChE
>

Re: PLEASE HELP regarding System Tap

[Khushboo Goel]

>> Is it possible i can know what address location is assigned to system
>> Tap buffer in the memory ??
>
> Well, yes, all of SystemTap is open source, so it's possible to peek
> into the internals.  The runtime sources contain the implementation for
> the maps data structures.  You can also run your script with "-p3" to
> see the code that's generated, and use that as a guide to tell how the
> maps are accessed.
>
> It seems very strange that you would ask for this though.  What would
> you do with the address once you find it?
>
 i want to write the data directly to that memory location instead of
system tap writing there , in that ways i won't get buffer overflow
problem, as i will be managing the data.

Anyways i will try -p3 and see how it helps me.
>

Re: PLEASE HELP regarding System Tap

[Frank Ch. Eigler]

Hi -

On Tue, Jul 08, 2008 at 05:48:59PM -0700, Khushboo Goel wrote:
> [...]
> I am actually sending some 10Gb or data over 8 tcp connection from 1 cpu ..
> But its post calculation so i don't think so i wana get into that ...

The overall quantity of network traffic should not matter.  The
histograms should work simply as counters of packet sizes.


> >> Is it possible i can know what address location is assigned to
> >> system Tap buffer in the memory ??
> >
> > There are many different buffers.  You might be interested in the one
> > the runtime can hand over via _stp_reserve(), but using that is not
> > routinely recommended.
> 
> Can you tell me more about it or any documentation on it ??? i want to
> go this runtime memory and write there directly.

This is unlikely to accomplish what you want.  It is an internal
interface, so there is no external documentation on it.


> I have observed one more thing , with the aggregate being in the code,
> and i just remove one of the embedded C code , i don't get any buffer
> overflow error, and it prints histogram and everything even for 20Gb
> of data .. with NAGLE off and sending 64 Bytes packet ( 96Bytes
> inclusive of header )
> Which is strange coz i expected aggregate to show overflow buffer error


We really need to see the exact error messages and the exact script
variants you are running, to help you better.  You're doing some good
work exploring these boundaries, but we really need the raw
information to help figure this out.


- FChE