From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 16277 invoked by alias); 19 Oct 2006 23:07:33 -0000 Received: (qmail 15819 invoked by uid 22791); 19 Oct 2006 23:07:31 -0000 X-Spam-Status: No, hits=-2.4 required=5.0 tests=AWL,BAYES_00,SPF_PASS X-Spam-Check-By: sourceware.org Received: from skunk.mtbrook.bozemanpass.com (HELO skunk.mtbrook.bozemanpass.com) (69.145.82.195) by sourceware.org (qpsmtpd/0.31) with ESMTP; Thu, 19 Oct 2006 23:07:28 +0000 Received: from [69.145.82.218] (unknown [69.145.82.218]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by skunk.mtbrook.bozemanpass.com (Postfix) with ESMTP id 1E1415583CD; Thu, 19 Oct 2006 16:07:27 -0700 (PDT) Message-ID: <45380531.10807@boreham.org> Date: Thu, 19 Oct 2006 23:07:00 -0000 From: David Boreham Reply-To: david_list@boreham.org User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Stone, Joshua I" CC: SystemTap Subject: Re: user mode backtrace References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2006-q4/txt/msg00200.txt.bz2 Stone, Joshua I wrote: >On Thursday, October 19, 2006 1:47 PM, David Boreham wrote: > > >>I'd like to get a stack trace for the process that made the >>system call I'm probing (I'm looking at filesystem access >>typically, so reads/writes/syncs etc). The systemtap backtrace >>function appears to only get the kernel mode stack which >>is not much use to me. I was wondering if anyone had >>discovered a good solution to this problem already ? >>I was thinking perhaps I could invoke pstack (gdb) >>on the current pid/tid. But I'm worried that doing so >>might deadlock since the process is inside a system >>call. >> >>I'm looking at a very large application that beats up on >>the filesystem, in case you're wondering why I want to do >>this. It's so large that nobody is quite sure what code >>access which files, when and why. >> >>Thanks. >> >> > >Deadlock issues aside, there's not really a way for you to invoke a >process (like pstack) from within a SystemTap script. You could run a >separate user program or script to do this for you though, and then you >just need to coordinate with SystemTap. Such a method might look like >this: > > Yeah, that was the kind of hack I was thinking of. >Of course at the end of the day, this is just a convoluted strace with a >stack printout. > Not really. The application that I'm battling with is resistent to my 'traditional' analysis techniques, which would include strace, gdb/pstack, oprofile etc. The big show stopper is that it isn't one big process that you can look at. It fires up numerous short lived processes. There isn't even one master process that exec's the subprocesses: some of them kick in on timers, watching the modification dates on files, and via local pipes. Hence the system-wide approach, hooking kernel activity and then looking upward to figure out what code in the application, in which process, is burning I/O resources.