From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 16948 invoked by alias); 21 Aug 2009 00:56:48 -0000 Received: (qmail 16941 invoked by uid 22791); 21 Aug 2009 00:56:47 -0000 X-SWARE-Spam-Status: No, hits=-1.7 required=5.0 tests=AWL,BAYES_00,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: sourceware.org Received: from mx1-old.redhat.com (HELO mx1.redhat.com) (66.187.233.31) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Fri, 21 Aug 2009 00:56:40 +0000 Received: from int-mx04.intmail.prod.int.phx2.redhat.com ([10.11.47.254]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id n7L0ua3d014407 for ; Thu, 20 Aug 2009 20:56:36 -0400 Received: from [10.16.2.46] (dhcp-100-2-46.bos.redhat.com [10.16.2.46]) by int-mx04.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id n7L0uWSd026419; Thu, 20 Aug 2009 20:56:32 -0400 Message-ID: <4A8DF197.2080107@redhat.com> Date: Fri, 21 Aug 2009 00:56:00 -0000 From: Masami Hiramatsu User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.1) Gecko/20090814 Fedora/3.0-2.6.b3.fc11 Thunderbird/3.0b3 MIME-Version: 1.0 To: Frederic Weisbecker CC: Ingo Molnar , Steven Rostedt , lkml , Ananth N Mavinakayanahalli , Avi Kivity , Andi Kleen , Christoph Hellwig , "Frank Ch. Eigler" , "H. Peter Anvin" , Jason Baron , Jim Keniston , "K.Prasad" , Lai Jiangshan , Li Zefan , =?UTF-8?B?UHJ6ZW15c8WCYXdQYXdlxYJjenlr?= , Roland McGrath , Sam Ravnborg , Srikar Dronamraju , Tom Zanussi , Vegard Nossum , systemtap , kvm , DLE Subject: Re: [TOOL] kprobestest : Kprobe stress test tool References: <20090813203403.31965.20973.stgit@localhost.localdomain> <4A847E30.9050903@redhat.com> <20090820184331.GA6078@nowhere> <4A8DA7CE.2040702@redhat.com> <20090821000108.GC6078@nowhere> In-Reply-To: <20090821000108.GC6078@nowhere> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2009-q3/txt/msg00422.txt.bz2 Frederic Weisbecker wrote: >> Most of them can be fixed just by adding __kprobes. >> Some of them which are already in the another section, kprobes >> should check the symbols are in the section. > > > You mean the blacklist? > > I also fear that putting bad kprobed functions into the kprobe > section or into the blacklist may hide some kprobe internal bugs. > > Doing so is indeed mandatory for functions that trigger tracing > recursion of things like that, but what if kprobe has an internal > bug that only triggers while probe a certain class of function. > > Ie: it would be nice to identify the reason of the crash for > each culprit in these lists. > > That may even help to find the others in advance. Indeed, actually I've found some bugs while making jump-optimization patches by using this stress test. But some of them are obviously what we just forget to add __kprobes, since those will be called from kprobes int3 handling functions. And also, many lock-related code has been changed. I think kprobes should use raw_*_lock, or prohibit to probe lock monitoring functions like lockdep, because it will cause recursive call. > > Also kprobes seems to be a very fragile feature (that's what > this selftest unearthes at least for me). > And it really needs a recursion detection that stops every kprobing > while reaching a given threshold of recursion. Something > that would dump the stack and the falling kprobe structure. Hmm, kprobes already has recursion detection(kp->nmiss), so maybe, we can check it. > > That would avoid such hard lockups and also help to identify > the dangerous symbols to probe. > > > >>> The problem is that I don't have any serial line in this >>> box then I can't catch any crash log. >>> My K7 testbox also died in my arms this afternoon. >>> >>> But I still have two other testboxes (one P2 and one P3), >>> hopefully I could reproduce the problem in these boxes >>> in which I can connect a serial line. >> >> Thank you for helping me to find it! >> >>> I've pushed your patches in the following git tree: >>> >>> git://git.kernel.org/pub/scm/linux/kernel/git/fgrederic/random-tracing.git \ >>> tracing/kprobes >>> >>> So you can send patches on top of this one. >> >> Great! I've found another trivial bugs, so I'll fix those on it. > > Cool :) > > Btw, here is the result of your stress test in a PIII (attaching the log > and the config). Thanks, I'll check that. Thank you, -- Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: mhiramat@redhat.com