From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 37706 invoked by alias); 31 Oct 2019 14:01:00 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 37677 invoked by uid 89); 31 Oct 2019 14:00:59 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-4.4 required=5.0 tests=AWL,BAYES_00,GIT_PATCH_2,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=ham version=3.3.1 spammy=worries, Enterprise, blind, bz X-HELO: mail-lj1-f174.google.com Received: from mail-lj1-f174.google.com (HELO mail-lj1-f174.google.com) (209.85.208.174) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 31 Oct 2019 14:00:48 +0000 Received: by mail-lj1-f174.google.com with SMTP id s4so6729561ljj.10 for ; Thu, 31 Oct 2019 07:00:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=noFAsOUOcd5YQViCRFNGZXqakpSzbnkLIpCHIYzJdVk=; b=sPDGWF4YWBavV4PAn4SyhfHpw0P/+IVkgwJzFTUh0Cuiu+Z0TUCCxNpoVvvncM5I2U bOkU9FB6G3tIDItFmAsMLr74Ksru9P4WbTzQgAUPB0nDY9eEi6xjdSdWr/U8A13dm6NG Mkc3DuRshsFppLWP7JoS56EHn/4UeUu6nNbf3dBgPiUqTs76Xz1RanXnlwyFsv5/kALY KvfEUWCh+KfinWL661XI0HJBDgZw4sHGMT2D9XA0v7v/whyjw5bWSmi3mXDZk/icmjo0 LPPaMcquHuWxOTsspHT3548MjbXi0+VhtYNy0135TfXqSEtoJfvtBAO4V2PljaaDRV9x Xuaw== MIME-Version: 1.0 References: <87bltyf0zp.fsf@redhat.com> In-Reply-To: <87bltyf0zp.fsf@redhat.com> From: Craig Ringer Date: Thu, 31 Oct 2019 14:01:00 -0000 Message-ID: Subject: Re: Newbie Notes To: "Frank Ch. Eigler" Cc: systemtap@sourceware.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-IsSubscribed: yes X-SW-Source: 2019-q4/txt/msg00022.txt.bz2 On Thu, 31 Oct 2019 at 04:10, Frank Ch. Eigler wrote: > > > > Q1: Can I make a probe conditional per-pid in the "if" clause? What > > *can* I do there? > > The [man stap] page covers probe conditionals this way: > > Probes may be decorated with an arming condition, consisting of a > simple boolean expres=E2=80=90sion on read-only global script vari= ables. > While disarmed (inactive, condition evaluates to false), some > probe types reduce or eliminate their run-time overheads. When > an arming condition evaluates to true, probes will be soon > re-armed, and their probe handlers will start getting called as > the events fire. [...] > > The key is that these conditions are -arming conditions-. They cannot > possibly use contextual values such as pid(), because in order to even > evaluate that condition, the probe would have to be armed & running! In > contrast, "boolean expressions on read-only global variables" may be > evaluated anywhere: namely at the conclusion of all the probes that > actually modify those variables. Then those probes can enqueue an > arm/disarm operation. > OK, that makes sense. I read over the manpage a few times, but I must've missed that. My apologies. > > Q2: Can I "return" or "exit" from a probe in a handler body or probe > > alias handler body? > > "next", as in awk. This too is in [man stap]. > Ah! Thanks. Right there in "STATEMENTS" too. Also in https://sourceware.org/systemtap/langref/6_Statement_types.html . Chalk one up to me being blind. My apologies. It might be handy to mention the keyword "return" or "exit" there for search convenience for similarly reading-ability-impaired folks. > Unused variables defined in aliases should not generate any warnings; or > maybe you're using "stap -u" ? > Nope, I noticed it with regular "stap". I'll re-check and see if I can tell what was going on and come back with details... OK. Take this script, which will work on anything you have lying around with simple adaptations: probe pg =3D process("postgres") { v1 =3D pid(); v2 =3D execname(); v3 =3D "availble for future callers"; } probe pg.function("PostgresMain") { printf("my pid is %d and name is %s\n", v1, v2); } probe pg.function("PostmasterMain") { printf("I don't use those variables\n"); } Since the variables defined here are in the prelude handler of the alias I wouldn't expect warnings for them, but on either stap 4.1 or 4.2-git-cadeeaa25 I get: sudo PATH=3D/usr/pgsql-11/bin:$PATH ./stap -v warnings.stp Pass 1: parsed user script and 476 library scripts using 201060virt/78756res/9736shr/69012data kb, in 150usr/30sys/176real ms. WARNING: Eliding assignment to 'v3': operator '=3D' at warnings.stp:5:12 source: v3 =3D "availble for future callers"; ^ WARNING: Eliding assignment to 'v1': operator '=3D' at :3:12 source: v1 =3D pid(); ^ WARNING: Eliding assignment to 'v2': operator '=3D' at :4:12 source: v2 =3D execname(); ^ WARNING: Eliding assignment to 'v3': operator '=3D' at :5:12 source: v3 =3D "availble for future callers"; ^ WARNING: Eliding side-effect-free expression : identifier 'v3' at :5:9 source: v3 =3D "availble for future callers"; ^ WARNING: Eliding side-effect-free expression : identifier 'v1' at :3:9 source: v1 =3D pid(); ^ WARNING: Eliding side-effect-free expression : identifier 'v2' at :4:9 source: v2 =3D execname(); ^ WARNING: Eliding side-effect-free expression : identifier 'v3' at :5:9 source: v3 =3D "availble for future callers"; ^ I'm inclined to suspect a severe case of PEBKAC is at work here, but it wouldn't be the first time I've found that when I speak up about something that confuses me it's confused others in the past too. > > Q4: tapscript backtraces from runtime faults, errors, etc, macro > > expanded-from for errors at runtime? > > -------------- > > > > Say I have some macro @MY_MACRO(x) or some function my_function(x). > > I do something in the macro that raises an error() or fault. > > The error from stap only reports the line number of the macro or > function. > > Aha. Hmm, this has come up before, but I don't think we recorded it > properly as an RFE (request-for-enhancement) BZ. It shouldn't be too > hard, given that we do track statement-by-statement where we are, in the > context->last_stmt variable. We would have to make that nestable, by > storing it inside the context->locals[LEVEL] struct instead. It's a bit > of work but not that much. Might you be interested in giving > implementing it a try? > Definitely interested. I'll have to check with work about getting the time since I'm currently rather short of outside-work coding time, but I think stap looks like a tool we could really benefit from. (I'm also getting into full-stack tracing with Jaeger and now OpenCensus, but that's much more high level and heavyweight, so both have a role). > @define @length1(a,v) %( @v =3D 0; foreach (_x in @a) { @v ++ } %) > @define @length2(a,v) %( @v =3D 0; foreach (_x,_y in @a) { @v ++ } %) > Thanks. Was mostly checking if I was missing something, a-la "next". > > Q6: is there any way to get the executable-path of the current pid() > > in a probe Just that. > > task_execname(task_current()) [man tapset::task] > Since posting, I found function::execname in tapset::context . It's not totally clear to me how that differs from using task_execname(task_current()), if at all. It'd be good to mention execname() in the beginners' guide alongside pid() and to see-also between those funcs. > > > Q7: Is there a string-concatenation operator that works at parse-time > > with macro-expansion? > > Say I want to use > > > > @define MY_PROG_BASEPATH %{ @1 %} > > @define MY_PROG(prog) %{ @MY_PROG_BASEPATH @prog %} > > Same rules as in C: adjacent string literals concatenate during parse. > The issue is specifically within @var and @cast expansion. It's done early, and doesn't appear to benefit from implicit string concatenation at all, macro-expanded or otherwise. Per my other post. This works: @define PGBIN %( @1 "/bin/postgres" %) probe begin { printf("%s\n", @PGBIN); } so does this: @define PGBIN %( @1 "/bin/postgres" %) probe process(@PGBIN).function("PostgresMain") { printf("%s\n", @PGBIN); } but not this (where $1 =3D /usr/pgsql-11/ ): @define PGBIN %( @1 "bin/postgres" %) function get_pgver:long() { return @var("server_version_num@guc.c", @PGBIN); } probe process(@PGBIN).function("PostgresMain") { printf("%s", @PGBIN, get_pgver()); } producing: parse error: expected ')' saw: string 'bin/postgres' at concat5.stp:1:29 source: @define PGBIN %( @1 ... bin/postgres ... "bin/postgres" %) ^ in expansion of macro: operator '@PGBIN' at concat5.stp:3:53 source: return @var("server_version_num@guc.c", @PGBIN); ^ 1 parse error. Pass 1: parse failed. [man error::pass1] however if I substitute the @var line with return @var("server_version_num@guc.c", "/usr/pgsql-11/bin/postgres"); it works. I haven't worked out exactly why yet, but it looks likely to be a matter of order-of-processing. I wonder if @var @cast etc need to be processed in a pass *after* user defined macros. > We don't have variadic macros, and variadic functions don't quite > propagate string literals well enough for this to work: > > function warn(msg,fmt,var1) { warn(sprint("%s " fmt, msg, var1)) } > function warn(msg,fmt,var1,var2) { warn(sprint("%s " fmt, msg, var1, > var2)) } > No worries, just figured it was worth checking. > > Q9: Can I iterate over a distinct-values slice of an array? > > ------- > > Say I have an array arr[x,y,z]. I want to iterate over all distinct > > "x" without repeats. > > There are no local associative array vars. > > Yeah, it'd require about the same amount of temporary storage as if you > spelled out the pair of operations consecutively: > > foreach ([x,y,z] in array) array2[x] =3D 2 > foreach ([unique] in array2) /* bob is unique */ ; > Right - but the docs say global array access requires locking and there are no local arrays. Isn't that a performance/overhead concern? It's not a big deal as the wonderful sorted-array-traversal operations let you just notice when there's a distinct key being visited. Just a touch verbose. Temporary arrays are a problem from a memory allocation perspective: > we just don't like doing it at run time, and arrays are too big to > put into the contect struct. > Makes sense, especially when there's all that concern about contiguous memory, fragmentation etc when working in the kernel. > Q10: How does the interpretation of probe bodies and function bodies > differ? > > ------- > > > > From what I've been able to tell, functions get expanded into probe > > bodies, somewhat macro-like. > > They could be, but actually they're translated to distinct blobs of C, > which are called/shared from multiple caller probes as needed. > Huh. I assumed they were expanded because it looked like when I called the same function from multiple probes, any warnings raised by the function got emitted repeatedly. I may be mistaken. It would be nice to document this alongside @cast and @var as it's very confusing as a new user to move something into a function and have its behaviour change suddenly. Come to think of it, wouldn't it be pretty easy to use the focus_on_module stuff to let functions be annotated with a default target module, or to define blocks with target modules? Think function get_application_name:string() default_module("postgres") { return @var("application_name@guc.c"); } or function get_application_name:string() { @default_module("postgres") { return @var("application_name@guc.c"); } } Pointless in the simple case above, but not in more complex deployments. OTOH, the lead time for anything like this to reach production systems is so long it's questionable whether it's worth bothering with as any scripts will need workarounds in the meantime. > > What can be done in probe bodies but not functions? There are clearly > > some differences since as my prior mail notes, functions resolve @cast > > and @var differently. > > That's becuase reusable functions don't have a natural context to > resolve types/$variables within. Probes do - the probe point itself. > Makes sense given what you said above; my assumption about functions was incorrect. And there'd be no way to infer the context from the call site without specializing the function into callsite-specific variants, which would be a nice feature to have but clearly not one that's present now. > > Q11: How to "fire" an "event" from a tapset? > > -------- > > [...] > > Is using aliases the wrong approach? If you want users to be able to > > say "run this handler when event y occurs", and event Y may happen 6 > > different ways with different relevant local variables etc each time, > > how do you do it? > > You may be overthinking it: just say probe y { ... } > declaratively. If you want to turn it on or off, use arming > conditionals or normal conditionals. > Won't work for me because I'm working in a complex multi-process context and I can't arm/disarm probes per-process. But I can just have my alias handler do if (foo) { next; } which is fine. > > Can a tapscript "fire" a synthetic probe? And put it in a context > > where it can access a process's globals etc? > > Nope, we cannot do something as messy as gdb's inferior-function-calls. > Yeah. No ptrace(), non-invasive, process is actively executing, working in kernel space, all that. I was thinking more of something like you have with associative arrays, synthetic probes on timer() events etc. Register an event (internally a global) at compile time. Fire it to trigger the handler. But it's not really necessary if probe aliases aren't meant to warn on unused vars and I can "next" from a probe alias handler. I'll confirm it works as expected in my test case... > Q12: How to make a probe or probe alias show up or not show up in > --monitor > > -------- > > --monitor mode seems to have some kind of magic for deciding what > > probes it shows hits for. > > So far I've only seen it produce output for SDT markers, not function > > probes, function return probes, etc. > > They should all show up, once they start getting hits. > Weird, they don't here. Will install latest stap and look into it; while I've studied the latest git sources I've still been testing against Systemtap translator/driver (version 4.1/0.177/0.176, rpm 4.1-2.fc30) > > How does it find out when a probe is hit and count it? Is there any > > way to suppress a probe as a "hit" within the body, given that per Q1 > > the probe conditional expression seems pretty limited. > > When you ask "how ...." - you can gird your loins and look at the > output of "stap -p3 ....". It'll show you exactly how. :-) > Thanks. I've spent a while browsing that output already. I guess I may've been unclear; I'm talking about how --monitor decides what is / isn't a hit. Does using "next" in a probe alias body suppress it? etc. As a heavy user of perf's "perf top", stap's --monitor is of great interest to me. I hadn't had time to read the monitor code before I posted this. I had a look now and AFAICS it's injecting code into the tapscript that probes procfs "monitor_status" to generate some json summary output about globals, probes, etc from a probe done in (). It calls __private___monitor_data_function_probes for each probe, which is defined in gen_monitor_data() in a bit of a wall of inline code as cstrings. That's embedded C (?) that looks up struct stap_probes which is defined by translate_pass() to create the macro STAP_PROBE_INIT(...) . I got a bit lost in the C++ around struct probe and struct statement at that point, but some guesswork took me to common_probe_entryfn_prologue and STP_TIMING which uses skipped_count(), but that seems to be session-level not per-probe. STP_ALIBI seems to generate a per-probe atomic_inc(...) but I didn't track that back to where it's defined or why. > > I didn't find much at all about --monitor mode yet. > > Yeah, it's not something we made a big fuss about. [man stap]. > > > > Q13: Easy way to handle enums? > > -------- > > [...] > > enum MyProgramEnum { > > XX, > > YY, > > ZZ > > }; > > > > manually to > > > > %define MyProgramEnum_XX %( 0 %) > > %define MyProgramEnum_XX %( 1 %) > > %define MyProgramEnum_XX %( 2 %) > > Have you seen @const()? It's basically a wrapper for embedded-C. > Yeah, I did see that, but you can't use embedded C normal in user tap scripts, only tapsets or guru mode, and as I mentioned I'm more than a tad nervous about trying to include/parse complex and portable headers like those in postgres. Plus that way you need the headers on hand. Not a biggie, it'd just be convenient to be able to copy the enums. > Some dwarf magic should let us find enum declarations in scope of a > probe, this sounds like a good RFE. > Huh, good point. I was only thinking parse/compile time, didn't occur to me it could be done at runtime. @enum(...) and if using -ggdb3 maybe even @macro(...) for simple integers. But I can't volunteer for that one, so it's likely to be an idle-for-a-long-time RFE unless I can get some allocated time for stap. > > Q14: Handling differences in target program version > > ------------ > > For the sake of an example, say that PostgreSQL version 11 has an > > argument / global variable "foo", and in PostgreSQL 12 it's gone, > > replaced with "foo_defined" and "foo_value". Details don't matter, > > point is that one symbol vanished, two different ones replace it. > > > > Is there any way to handle these version differences by making probes > > conditional on target expressions/values? E.g. "use this probe if > > @var("pg_version_num") > 120000, otherwise this other probe" ? > > A probe handler can use @defined() and macro wrappers to adapt to > the presence or absence of context variables. See our old pal > [man stap] and its friends [man stapprobes]. Many tapsets use > it too. > > PR13009 would let probe point conditionals include @defined() tests > (which normally resolve to literals early during translation). > > > > Q15: variable/dynamic probes > > ---------- > > > > Say I want to probe a bunch of different binaries within some basedir > > I want to pass as a script argument (or better yet, resolve from the > > PATH of a target binary). > > > > It seems I can't just > > alias pg.backend =3D process(@1 "/bin/myproc"); > > This works for me: > > stap -e ' > probe pg.backend =3D process(@1 "/bin/myproc") {} > probe pg.backend.function("foo") {} > ' /directory/prefix > > > > but because string literal auto-concatentation doesn't happen for > > macro-expanded strings > > If so, this is a bug. > > > > New user experience, docs suggestions > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > > > > > Document stringifying numbers > > ---------- > > > > Concatenation with . doesn't stringify numbers. > > Yup, because "." forms an important signal to the type inferences > that its operands are strings. > Which makes sense. There are many extremely useful hints in systemtap already and this might be a candidate for another though: $ sudo stap -e 'probe begin { a =3D 1; b =3D "b" . a; printf("b is %s\n= ", b); }' semantic error: type mismatch (string): identifier 'a' at :1:32 source: probe begin { a =3D 1; b =3D "b" . a; printf("b is %s\n= ", b); } ^ semantic error: type was first inferred here (long): identifier 'a' at :1:15 source: probe begin { a =3D 1; b =3D "b" . a; printf("b is %s\n= ", b); } for operator(.) like "operator '.' takes string operands, do you need sprint() or sprintf()? see [function::sprint], [function::sprintf]" ? Unsure how simple it is to determine the operator (.) is the cause of the type mismatch though and frankly it's probably easier to just add a note alongside the . operator in the docs pointing to sprint() and sprintf() for coercing integers to strings. A note on the print/sprint entry in the manpage that mentions the keyword "integer" wouldn't hurt either. I'll see if I can put together a docs PR soon. > > Array wildcards, iterate-with-return need to be documented in BG > > ----------- > > > > I found systemtap infinitely easier to use once I stumbled across the > > array wildcard support. This would've been immensely useful early on > > and I think it should be highlighted in the beginners' guide: > > [...] > > I think we're suffering to some extent because we have too much > documentation - and not all of it is updated when some new feature > gets added. Right, I get that. You've pointed out places where things were documented and I didn't find them because I was doing targeted searches and reading key sections, having not read it cover-to-cover yet. I'm prone to writing too much documentation that people don't read myself. > > Document that $1 ... $n and $# are usable in preprocessor > > --------- > > > > I tried to write code that used $# to decide whether $2, etc were > > valid and got very confused that stap kept spamming warnings. [...] > > OK. There is also the tapset/argv.stp file that defines a boringly > C-like argc & argv[] global variable pair. > Huh, handy. That's good to know :) . TBH I probably would've assumed the argv tapset was for target arguments anyway had I seen it ; it's worth x-ref'ing where $1 @1 etc are doc'd in man stap. > > Document that probe definitions cannot reference variables > > ----------- > > > > When getting started, it's surprising that you cannot > > > > pgbasedir =3D "/path/to/postgres"; > > alias pg.backend =3D process($pgbasedir . "/bin/postgres") > > and it might be nice to cover what you can and cannot do in probe > arguments. > > Probe points must be literals, as they must be evaluated at translate > time. Therefore they cannot vary. > Yeah, I get that now :) . This is one of those "confused me at the beginning" things. I didn't know about macros yet. > No process-local / thread-local vars > > ------------ > > Any non-probe-scoped variable, whether "global" or "private", is > > global to the whole tapscript. There's AFAICS no way to instance such > > variables per-process or per-thread automatically. So the beginners > > guide should mention this and suggest the pattern: [...] > > See the "this.stp" tapset for syntactic sugar for "hidden array > indexed with tid()", like: > > probe tp_syscall.read { > @this1 =3D buf_uaddr > } > probe tp_syscall.read.return { > buf_uaddr =3D @this1 > printf("%s", user_string(buf_uaddr)) > } > Yeah, I found that, but there's no equivalent for pid() .... ugh, and I'm so stupid it hurts. For non-threaded processes pid() corresponds 1:! to tid() . Sigh. > > SDT probe data types and argument names: > > ----------- > > > > SDT probes have arguments $arg1... $argn, and no data type information > > is preserved. That's exceedingly unfortunate as they're otherwise a > > super useful tool. Is it feasible to do anything like embed the param > > names and types in the probes ELF section or an extension section of > > it, for reading by stap? > > Tough as from a C macro call given values, we can't get type names as > literals that we can just include into the .sdt elf note. Will think > about it. > Right, not saying it's easy ;) DWARF with -ggdb3 has the macro definition, but probably not in a form convenient to map through the layers of expansion back to the original argument types. I think it's just the raw definition. I was thinking about it mostly for SDTs defined by probes.d, which already syntatically accepts c-alike probe definitions, it just ignores the argument names and types completely AFAICS. In this case the dtrace script could generate a second macro alongside the DTRACE_PROBE2 or whatever like a STAP_PROBE_ARG_NAMES2("somearg", "someotherarg"); STAP_PROBE_ARG_TYPES2("const char *", "uint32_t"); which can presumably be inserted into a custom ELF section in much the same manner as the probe points themselves are generated. Whether it's useful to do so is another matter, though, as it'd require other tools to agree upon and adopt the scheme. Also, this way the type information would only be strings, which limits options for type checking and for doing anything useful with it in the face of typedefs etc. Dude, thanks a lot for all this info. It's worth virtual gold to hear > from users with fresh eyes. I hoped so. I know it can also be frustrating because to you it's "just right there dammit!" - but new users face information overload, are trying to adapt to new terminology and structures, and are usually under time pressure. So it's not just whether it's documented, but where and how. I wish I could do better at that myself. Sometimes I feel like the only user of some of the docs I write is ... myself. > For those cases where some missing feature > or a bug was identified (as opposed to RTFManPage), it would be great to > get them into the queue (sourceware.org/bugzilla), and we can most > certainly take patches, even little ones like your preferred wordings > for the documentation nits. > > If you'd like me to open bugzilla's for my impression of them, can do, > but doing it yourself would get you into the system and get the request > & sample-desire worded just right. > Can do. I'll try to prep the odd patch as well. --=20 Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise