From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 2793 invoked by alias); 8 Jul 2015 14:12:51 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 2753 invoked by uid 48); 8 Jul 2015 14:12:47 -0000 From: "dsmith at redhat dot com" To: systemtap@sourceware.org Subject: [Bug runtime/18642] Probes not always invoked when probed functions are called Date: Wed, 08 Jul 2015 14:12:00 -0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: systemtap X-Bugzilla-Component: runtime X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: dsmith at redhat dot com X-Bugzilla-Status: WAITING X-Bugzilla-Resolution: X-Bugzilla-Priority: P2 X-Bugzilla-Assigned-To: systemtap at sourceware dot org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: http://sourceware.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-SW-Source: 2015-q3/txt/msg00018.txt.bz2 https://sourceware.org/bugzilla/show_bug.cgi?id=18642 David Smith changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |WAITING CC| |dsmith at redhat dot com --- Comment #1 from David Smith --- (In reply to Izi Anavi from comment #0) > Created attachment 8421 [details] > stap-report output > > Systemtap probes are invoked only some of the time when the probed function > is called. > > This happens in two scenarios which may or may not be related: > > 1. Scenario I: > System calls such as read, write or bind using probes syscall.* (e.g.: > syscall.read, syscall.write, syscall.bind) are not called consistently. > > In contrast, when using process.syscall probe, the probes are called > consistently when the probed functions are called. > > We can't understand the discrepancy between the probe types behavior. I'm not sure this is 100% of your problem, but certainly glibc can be a culprit. Take open() for example. Based on how you call it and what architecture you are on, a user program's call to open() might end up actually calling (one or more) of the following syscalls: creat, open, or openat. Try comparing systemtap's output to strace's output to take glibc out of the picture. Do something like this (once you've installed the systemtap testsuite - otherwise get sys.stp from here ): # strace foo.exe # stap /usr/share/systemtap/testsuite/systemtap.syscall/sys.stp -c foo.exe The syscalls seen by strace and the ones seen by systemtap should match up (once the executable gets going - systemtap sees earlier syscalls than strace does). > 2. Scenario II: > Probe for process creation using probe kernel.function("sys_execve") and > user space probes for the created process (and its subprocesses) are not > called consistently. In our system the created process is usually bash, and > the user space probes on bash - > probe process("/bin/bash").function("pre_process_line") > probe process("/bin/bash").function("read_command") > > are not invoked either when process creation probe isn't invoked. If I'm reading this one correctly, you aren't probing widely enough. Instead of probing kernel.function("sys_execve"), you should be probing syscall.execve, syscall.compat_execve, syscall.execveat, and syscall.compat_execveat. Once again compare strace's output to systemtap's on a target executable. Another thing to try would be a process.begin probe and see if that is missing process creation also. Are you sure the target process is doing an exec in all cases? As far as missing process.function user-space probes go, I'm unsure what is going on. We probably need to clear up some of the earlier errors first. > It is worth mentioning that in these scenarios we run systemtap over an lxc > container and all the relevant processes run in this container. Hmm. If possible, try to run the same scenarios on the host system and see what you get. > Attached is the output from stap-report. We run systemtap 2.7 over ubuntu > 14.04.2 (kernel 3.13.0-24-generic). Those are recent enough versions of systemtap and kernel where this should work correctly. -- You are receiving this mail because: You are the assignee for the bug.