From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 78819 invoked by alias); 24 Apr 2017 11:59:03 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 78797 invoked by uid 89); 24 Apr 2017 11:59:02 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.1 required=5.0 tests=AWL,BAYES_00,RP_MATCHES_RCVD,SPF_PASS autolearn=ham version=3.3.2 spammy=presents, H*MI:corp, Hx-spam-relays-external:14.3.319.2, H*RU:14.3.319.2 X-HELO: mga07.intel.com Received: from mga07.intel.com (HELO mga07.intel.com) (134.134.136.100) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 24 Apr 2017 11:59:01 +0000 Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga105.jf.intel.com with ESMTP; 24 Apr 2017 04:59:01 -0700 X-ExtLoop1: 1 Received: from irsmsx102.ger.corp.intel.com ([163.33.3.155]) by orsmga004.jf.intel.com with ESMTP; 24 Apr 2017 04:59:01 -0700 Received: from irsmsx111.ger.corp.intel.com (10.108.20.4) by IRSMSX102.ger.corp.intel.com (163.33.3.155) with Microsoft SMTP Server (TLS) id 14.3.319.2; Mon, 24 Apr 2017 12:59:00 +0100 Received: from irsmsx107.ger.corp.intel.com ([169.254.10.107]) by irsmsx111.ger.corp.intel.com ([169.254.2.58]) with mapi id 14.03.0319.002; Mon, 24 Apr 2017 12:59:00 +0100 From: "O Mahony, Billy" To: "systemtap@sourceware.org" Subject: reducing cost of user-space probes Date: Mon, 24 Apr 2017 11:59:00 -0000 Message-ID: <03135AEA779D444E90975C2703F148DC2F8B6BED@IRSMSX107.ger.corp.intel.com> x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOGEyZGU2NTAtM2IyZC00NTUzLWIyYmMtNzQ5NGMwZTFkMzJjIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6Im1nQklZQ3VSeDlJaElXRk02RG5lY1wvVWNzaTZVbzBcL3hXWmhqZWhxb1wvb289In0= x-ctpclassification: CTP_IC dlp-product: dlpe-windows dlp-version: 10.0.102.7 dlp-reaction: no-action Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-SW-Source: 2017-q2/txt/msg00030.txt.bz2 Hi, I'm new to systemtap and I am using it to add some probes into a user space= application. The probe is pretty simple - it collects one integer argument and presents = a histogram every 3 seconds. The probe is working fine and I'm getting results that are sensible. The ap= plication is a packet processing application that is using a user space io = library (DPDK) to read batches of network packets directly into user space.= The probe is called about 750K times per second (I have 10Gb link with 6= 4B packets which generates 14.8M packets per second - but the batch size (t= hat's the stat I'm tracing) - is about 20 so 750K probe hits per sec.=20 When the probe is in use I see less performance from the packet processing = application - it starts loosing packets at about 90% of it's non-probed thr= oughput.=20 However, when I run stap I see: > Pass 4: compiled C into "stap_13723.ko" in 9020usr/980sys/10638real ms Does this mean that each time the probe is hit that a system call is made t= o this new .ko module? That would surely mean quite a lot of overhead. If t= his is correct, can this overhead be avoided for user space probes. Alternatively is there a way to only execute the script every n times the p= robe is hit? Maybe there is a compile time macro that does this or some .stap command th= at does an early return from the script X% of the time. I searched for 'sam= ple/sampling' in the lang ref but I didn't see anything. Thanks for any help you can give. Billy