From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 6667 invoked by alias); 23 Jun 2011 10:12:39 -0000 Received: (qmail 6658 invoked by uid 22791); 23 Jun 2011 10:12:38 -0000 X-SWARE-Spam-Status: No, hits=-6.8 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,SPF_HELO_PASS,T_RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Thu, 23 Jun 2011 10:12:19 +0000 Received: from int-mx02.intmail.prod.int.phx2.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p5NACJCF004928 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Thu, 23 Jun 2011 06:12:19 -0400 Received: from localhost (vpn1-5-79.ams2.redhat.com [10.36.5.79]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id p5NACI5h017666; Thu, 23 Jun 2011 06:12:18 -0400 Date: Thu, 23 Jun 2011 10:12:00 -0000 From: "Richard W.M. Jones" To: Josh Stone Cc: systemtap@sourceware.org Subject: Flushing systemtap output without restarting (was: Re: Rapidly running systemtap causing hangs or oops) Message-ID: <20110623101217.GH18438@amd.home.annexia.org> References: <20110622230025.GG18438@amd.home.annexia.org> <4E028028.4010603@redhat.com> <20110623075126.GJ803@amd.home.annexia.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110623075126.GJ803@amd.home.annexia.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2011-q2/txt/msg00320.txt.bz2 On Thu, Jun 23, 2011 at 08:51:26AM +0100, Richard W.M. Jones wrote: > Unfortunately for the real program I'm writing, I really do need a way > to box stap around each test. The problem I was having before was > that there was quite a long delay between my test running and stap > probes firing (or at least, seeing stap output). I need the stap > output from one test to be clearly distinct from the stap output from > the next test. If there was a way to run the test and then say to > stap "now flush all your output" before running the next test, then > that would be acceptable. > > I thought about using the process ID, but ideally my tests will all > run as the same pid. My first idea for "flushing" stap output involves putting a timestamp at the start of each stap line, then matching up those timestamps with when the tests were run. In my test I have: for pass in `seq 1 1000`; do d=$(date +'%s.%N') echo $d: mount mount -o loop /tmp/fs /tmp/mnt umount /tmp/mnt sync done In my probe I have: probe module("ext2").statement ("*@*.c:*") { us = gettimeofday_us(); printf ("%d.%d: %s\n", us/1000000, us%1000000, pp()); } My aggregated output is supposed to look like this: 1308822712.884907138: mount 1308822712.920008: module("ext2").statement("ext2_mount@fs/ext2/super.c:1378") 1308822712.920015: module("ext2").statement("ext2_mount@fs/ext2/super.c:1379") [etc] Unfortunately this doesn't work. It seems like the probes don't run synchronously. Are they running asynchronously in a bottom half / softirq / {whatever Linux is calling them this week}? It certainly looks like they run asynch, because my actual output (sometimes) looks like: 1308822714.015994546: mount 1308822714.424888519: mount 1308822714.43626: module("ext2").statement("ext2_mount@fs/ext2/super.c:1378") 1308822714.43634: module("ext2").statement("ext2_mount@fs/ext2/super.c:1379") 1308822714.442957: module("ext2").statement("ext2_mount@fs/ext2/super.c:1378") 1308822714.442965: module("ext2").statement("ext2_mount@fs/ext2/super.c:1379") The output from two tests is mixed together, and the timestamps aren't accurate. That's no good for the testing I had in mind. So is there a way to flush systemtap output without restarting it? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top