Newbie Notes

Newbie Notes

[Craig Ringer]

Hi all

I've spent the last week or so getting into SystemTap, which was on my
"to learn" list for quite some time. I'm amazed by how powerful it is.
I also know that feedback from people who are learning my own software
is useful, so I thought I'd share some of the challenges and
roadblocks I experienced along the way and some suggestions for docs
etc.

I also have some open questions I'd love some suggestions on. I'll
lead with them, then get into my notes on things I stumbled over when
learning. For context, I'm writing probes against PostgreSQL, which is
a multi-processing fork() based server, so I'm pretty much always
interested in the state of many processes and their interactions.

Q1: Can I make a probe conditional per-pid in the "if" clause? What
*can* I do there?
----

Say I have some associative array "interesting_procs" and I don't want
to see "handle_some_event()" calls for any other PIDs.

I expected

  probe process("myapp").function("handle_some_event()") if (pid() in
interesting_procs) {
    ....
  }

to work, but pid() isn't allowed in a probe condition. So you can't
seem to use globals to toggle probes on a finer scope than
whole-script-globally.

The docs don't really explain what you can and cannot do in a probe if
condition.



Q2: Can I "return" or "exit" from a probe in a handler body or probe
alias handler body?
----

Assuming there's no good answer to Q1, is there any way to "return"
early from a probe body? Especially from a probe alias handler, so the
alias can cause the main probe body to be skipped entirely?

I know I can set a variable in the alias body, then test it in the
main probe body, but that's ugly and cumbersome. Is there a better
way?

e.g if I have:

probe myproc.something = probe process("myproc").function("something") {
    is_interesting = (@var("foo"));
};

probe myproc.something {
    if (is_interesting) {
        printf("XXX....");
    }
}


I'd like to be able to  instead


probe myproc.something = probe process("myproc").function("something") {
  if (@var("foo") == 0)
    return;
};

probe myproc.something {
    printf("XXX....");
}


Especially since the unused variables warnings stap emits mean that
probe alias handlers that define variables tend to create huge amounts
of warnings spam if some users of the alias don't care about all the
variables.

Q3: Some way to mark individual variables to suppress unused warnings?
---------------

In probe alias bodies it's often useful to define a set of variables
for use by the probe(s) that use the alias. But stap likes to spam
warnings about these if not all of them are used by all probes.

-w is a very heavy hammer to use for this as it hides many legitimate
issues too.

Is there a way to suppress unused warnings only for alias bodies? Or
preferably, mark individual variables as "don't warn if unused"?


Q4: tapscript backtraces from runtime faults, errors, etc, macro
expanded-from for errors at runtime?
--------------

Say I have some macro @MY_MACRO(x) or some function my_function(x).

I do something in the macro that raises an error() or fault.

The error from stap only reports the line number of the macro or function.

Is there any way to see where it was called from too? Or, for macro
expansions, to show the "expanded from" text for runtime errors like
are produced in compile-time errors?

Without this, tracking down a fault arising from some widely used
macro or function is ... painful.

For functions using ubacktrace() etc sometimes helps; it doesn't tell
you anything about the stap state but it offers hints about what probe
might've called the function. But that's a pretty backwards way of
doing it.


Q5: is there any way to get the len()/size() of a non-statistical array?
-------

Say I have a non-statistical associative array "arr". Can I get a
membership count with anything other than

    n = 0;
    foreach ([x] in arr) { n++; };
    printf("length is %d\n", n);

Q6: is there any way to get the executable-path of the current pid() in a probe
--------

Just that.

Q7: Is there a string-concatenation operator that works at parse-time
with macro-expansion?
--------

Say I want to use

   @define MY_PROG_BASEPATH %{ @1 %}
   @define MY_PROG(prog) %{ @MY_PROG_BASEPATH @prog %}

Q8: nicer way to warn() or error() with params than sprintf?
--------

If I have a warn(), error() or assert() I want to include the values
of variables in, is there a nicer way to do it than assert(condition,
sprintf(...)) ?

Especially for assert.

Q9: Can I iterate over a distinct-values slice of an array?
-------

Say I have an array arr[x,y,z]. I want to iterate over all distinct
"x" without repeats.

There are no local associative array vars.

AFAICS I have to

last_x = 0;
for (arr = [x+,y,z] in arr) {
  if (x != last_x) {
    printf("found distinct x: %d\n", x);
    last_x = x;
  }
}

which is kinda clunky, especially as I don't have much of a way to
wrap it up in a generic/reusable way.

Is there a better way?

Q10: How does the interpretation of probe bodies and function bodies differ?
-------

From what I've been able to tell, functions get expanded into probe
bodies, somewhat macro-like.

How do functions and macros differ? What can be done in probe bodies
but not functions? There are clearly some differences since as my
prior mail notes, functions resolve @cast and @var differently.

Q11: How to "fire" an "event" from a tapset?
--------

Say I have some complex condition or event that doesn't lend itself
well to being written as a probe alias. Especially with the matters in
Q1, Q2 and Q3 making it hard to present the main handler for the alias
with a consistent interface where it's only invoked at the right
times, and always with the same vars defined.

Is using aliases the wrong approach? If you want users to be able to
say "run this handler when event y occurs", and event Y may happen 6
different ways with different relevant local variables etc each time,
how do you do it?

Can a tapscript "fire" a synthetic probe? And put it in a context
where it can access a process's globals etc?


Q12: How to make a probe or probe alias show up or not show up in --monitor
--------

--monitor mode seems to have some kind of magic for deciding what
probes it shows hits for.

So far I've only seen it produce output for SDT markers, not function
probes, function return probes, etc.

How does it find out when a probe is hit and count it? Is there any
way to suppress a probe as a "hit" within the body, given that per Q1
the probe conditional expression seems pretty limited.

I didn't find much at all about --monitor mode yet.


Q13: Easy way to handle enums?
--------

Assume that project X has complex headers that don't lend themselves
well for inclusion in an embedded-C tapset, you're more than a bit
nervous about doing so for a large/complex project, you want to handle
different versions of the project without the user having to dig out
the right headers, etc etc.

Is there any better way to define individual enums in a tapset script
than translating

enum MyProgramEnum {
   XX,
   YY,
   ZZ
};

manually to

%define MyProgramEnum_XX %( 0 %)
%define MyProgramEnum_XX %( 1 %)
%define MyProgramEnum_XX %( 2 %)

I'm aware of the @cast header-inclusion support, but it makes me
exceedingly nervous to use it. I'd probably land up using gdb script
to generate faked-up headers with just the enum definitions or
something, but that's pretty horrible too...


Q14: Handling differences in target program version
------------

For the sake of an example, say that PostgreSQL version 11 has an
argument / global variable "foo", and in PostgreSQL 12 it's gone,
replaced with "foo_defined" and "foo_value". Details don't matter,
point is that one symbol vanished, two different ones replace it.

Is there any way to handle these version differences by making probes
conditional on target expressions/values? E.g. "use this probe if
@var("pg_version_num") > 120000, otherwise this other probe" ?

Or is it necessary to use %( conditionals %) and have the user specify
the inputs as commandline arguments?

AFAICS you can't try{}catch{} a @var expansion, etc as the're
compile-time issues not runtime faults.

Q15: variable/dynamic probes
----------

Say I want to probe a bunch of different binaries within some basedir
I want to pass as a script argument (or better yet, resolve from the
PATH of a target binary).

It seems I can't just

    alias pg.backend = process(@1 "/bin/myproc");

I can:

   @define PGBIN %( @1 %)
   alias pg.backend = process(@PGBIN)

but because string literal auto-concatentation doesn't happen for
macro-expanded strings, I cannot then:

   @define PGBASEDIR %( @1 %)
   alias pg.backend = process(@PGBASEDIR "/bin/postgres")

so I'm yet to find a good way to handle things like paths for process
probes. Ideas?

Is this not possible without some kind of not-currently-extant
operator for parse-time concatenation of macro-expanded strings?



New user experience, docs suggestions
===============================


Document stringifying numbers
----------

Concatenation with . doesn't stringify numbers.

Document that you just use sprintf to do it, or you may want to use
the "println" statement that automatically handles concatenation and
stringification.



Array wildcards, iterate-with-return need to be documented in BG
-----------

I found systemtap infinitely easier to use once I stumbled across the
array wildcard support. This would've been immensely useful early on
and I think it should be highlighted in the beginners' guide:

private arr;

arr[pid(),"foo"] = 42;

for (val = [pid, ph] in arr) {
}

for (val = [pid, ph] in arr[pid(),*]) {
}

for (val = [pid, ph] in arr[*,42]) {
}

if ([*,*] in arr) {
 /// true if array non-empty
}

if ([pid(),*] in arr) {
  // true if any entry for pid() in array
}

delete arr[pid(), *];

delete arr[*,*];


Document that $1 ... $n and $# are usable in preprocessor
---------

I tried to write code that used $# to decide whether $2, etc were
valid and got very confused that stap kept spamming warnings.

It seems you're supposed to use these in conditionals? Like

        myvar = %( $# >2 %? @3 %: "" %)

or equivalent conditionally-compiled blocks.

This could use some docs where script-args are mentioned and in the
conditional compilation section, also maybe a hint when stap warns of
unused arguments during compilation.


Document that probe definitions cannot reference variables
-----------

When getting started, it's surprising that you cannot

    pgbasedir = "/path/to/postgres";
    alias pg.backend = process($pgbasedir . "/bin/postgres")

and it might be nice to cover what you can and cannot do in probe arguments.


No process-local / thread-local vars
------------

Any non-probe-scoped variable, whether "global" or "private", is
global to the whole tapscript. There's AFAICS no way to instance such
variables per-process or per-thread automatically. So the beginners
guide should mention this and suggest the pattern:

    some_global[pid()] = "value";

and

   if ( pid() in some_global )

and

   my_some_global = some_global[pid()];

and warn that users should add a probe that deletes the value when the
pid exits so that the array doesn't full up, e.g.

probe process("myproc").end {
  delete some_global[pid()];
}


Of course having syntax to wrap this or even better, having lockless
thread-local/process-local variables, would be way better.



SDT probe data types and argument names:
-----------

SDT probes have arguments $arg1... $argn, and no data type information
is preserved. That's exceedingly unfortunate as they're otherwise a
super useful tool. Is it feasible to do anything like embed the param
names and types in the probes ELF section or an extension section of
it, for reading by stap?

Alternately, though not as good, would be for the 'dtrace' tool to
generate a tapset with aliases and @casts from the .d file. I may
write a script for this; if I do, I'll post it here.


String access: user_string, user_string_warn
-------------

It'd be good to make these more prominent, most importantly in the
target variables section of the beginners guide
https://sourceware.org/systemtap/SystemTap_Beginners_Guide/targetvariables.html

Also warn about the @var caveat my immediately prior mail mentioned,
re needing absolute paths for modules and not doing PATH-expansion
when using @var from a function().

Target array access
-------------

It'd be nice if the docs mentioned how to access target array
variables or array-valued struct members. It's fairly obvious for
locals:

    $targetvar[index];

    $targetstruct->arraymember[index];

For @var and @cast it works similarly, but will result in confusing
failures to resolve the variable or type if you do it wrong. Parens
are allowed and seem to be useful (needed?). E.g.

    (@var("globalarray"))[idx]

    (@var("globalstruct")->arraymember)[idx]

Deref'ing scalar as array produces confusing symbol not found error
-------

but if you attempt to

    @var("globalstruct")[0]

stap will report that it cannot find the symbol "globalstruct" even
though it's right there. The real error seems to be that it can't cope
with the attempt to dereference it as an array when it's a scalar.

e.g. if globalstruct is

struct globalstructtype {
     my_type varlen_array_member[0];
};

globalstructtype *globalstruct;

then forgetting to reference the member and using

    @var("globalstruct")[0]

produces a symbol resolution error for globalstruct, which is confusing.


Typecasting
-----------

Same warning for @cast as for @var re functions, module paths.

Warn that @cast to a pointer-hiding typedef will produce confusing
results, strange faults, etc.


Queue stats
-------------


tapset crosslinking
---------------

It'd be nice to see more x-linking of important tapset info in the
main docs and beginner docs, e.g. linking to




-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Newbie Notes

[Craig Ringer]

Oh, another tip I wanted to add to my suggestions for things to put in
the new user docs:

If you get a fault on a $local or a @var("global") check that the
target type is wide enough. I've found that in some cases I have to
user_int8(&$local) or user_int8(&@var("global")) to read an int8
target variable.

I don't really understand why this appears to be necessary at some
times but not others, etc, but there you go.

Re: Newbie Notes

[Frank Ch. Eigler]

Hi, Craig -

First of all, thanks for your kind words on lwn.


craig wrote:

> [...] I thought I'd share some of the challenges and
> roadblocks I experienced along the way and some suggestions for docs
> etc.

Great.

> Q1: Can I make a probe conditional per-pid in the "if" clause? What
> *can* I do there?

The [man stap] page covers probe conditionals this way:

       Probes may be decorated with an arming condition, consisting of a
       simple boolean expres‐sion on read-only global script variables.
       While disarmed (inactive, condition evaluates to false), some
       probe types reduce or eliminate their run-time overheads.  When
       an arming condition evaluates to true, probes will be soon
       re-armed, and their probe handlers will start getting called as
       the events fire.  [...]

The key is that these conditions are -arming conditions-.  They cannot
possibly use contextual values such as pid(), because in order to even
evaluate that condition, the probe would have to be armed & running!  In
contrast, "boolean expressions on read-only global variables" may be
evaluated anywhere: namely at the conclusion of all the probes that
actually modify those variables.  Then those probes can enqueue an
arm/disarm operation.


> Q2: Can I "return" or "exit" from a probe in a handler body or probe
> alias handler body?

"next", as in awk.  This too is in [man stap].


> [...]
> I know I can set a variable in the alias body, then test it in the
> main probe body, but that's ugly and cumbersome. Is there a better
> way?
> [...]
> I'd like to be able to  instead
>
> probe myproc.something = probe process("myproc").function("something") {
>   if (@var("foo") == 0)
>     return;
> };

Say "next" instead of "return", and bob is your uncle.


> Especially since the unused variables warnings stap emits mean that
> probe alias handlers that define variables tend to create huge amounts
> of warnings spam if some users of the alias don't care about all the
> variables.

Unused variables defined in aliases should not generate any warnings; or
maybe you're using "stap -u" ?


> Q3: Some way to mark individual variables to suppress unused warnings?
> ---------------
>
> In probe alias bodies it's often useful to define a set of variables
> for use by the probe(s) that use the alias. But stap likes to spam
> warnings about these if not all of them are used by all probes.

That's not normal behaviour.  One point of aliases is to define a whole
menu of potentially useful variables, which a probe may quietly choose
from.  If that's not working, please report.


> Q4: tapscript backtraces from runtime faults, errors, etc, macro
> expanded-from for errors at runtime?
> --------------
>
> Say I have some macro @MY_MACRO(x) or some function my_function(x).
> I do something in the macro that raises an error() or fault.
> The error from stap only reports the line number of the macro or function.

Aha.  Hmm, this has come up before, but I don't think we recorded it
properly as an RFE (request-for-enhancement) BZ.  It shouldn't be too
hard, given that we do track statement-by-statement where we are, in the
context->last_stmt variable.  We would have to make that nestable, by
storing it inside the context->locals[LEVEL] struct instead.  It's a bit
of work but not that much.  Might you be interested in giving
implementing it a try?


> Q5: is there any way to get the len()/size() of a non-statistical array?

There should be.  As you realize, statistics arrays are more complicated
(since they are per-cpu, until a merging mandated by a foreach()
iteration).

Until the language/runtime has an optimized version, you can still
at least make the call sites small via macros such as:

@define @length1(a,v) %( @v = 0; foreach (_x in @a) { @v ++ } %)
@define @length2(a,v) %( @v = 0; foreach (_x,_y in @a) { @v ++ } %)


> Q6: is there any way to get the executable-path of the current pid()
> in a probe Just that.

task_execname(task_current())   [man tapset::task]


> Q7: Is there a string-concatenation operator that works at parse-time
> with macro-expansion?
> Say I want to use
>
>    @define MY_PROG_BASEPATH %{ @1 %}
>    @define MY_PROG(prog) %{ @MY_PROG_BASEPATH @prog %}

Same rules as in C: adjacent string literals concatenate during parse.


> Q8: nicer way to warn() or error() with params than sprintf?
> --------
> If I have a warn(), error() or assert() I want to include the values
> of variables in, is there a nicer way to do it than assert(condition,
> sprintf(...)) ?

We don't have variadic macros, and variadic functions don't quite
propagate string literals well enough for this to work:

function warn(msg,fmt,var1) { warn(sprint("%s " fmt, msg, var1)) }
function warn(msg,fmt,var1,var2) { warn(sprint("%s " fmt, msg, var1, var2)) }


> Especially for assert.
>
> Q9: Can I iterate over a distinct-values slice of an array?
> -------
> Say I have an array arr[x,y,z]. I want to iterate over all distinct
> "x" without repeats.
> There are no local associative array vars.

Yeah, it'd require about the same amount of temporary storage as if you
spelled out the pair of operations consecutively:

    foreach ([x,y,z] in array)  array2[x] = 2
    foreach ([unique] in array2) /* bob is unique */ ;

> AFAICS I have to
>
> last_x = 0;
> for (arr = [x+,y,z] in arr) {
>   if (x != last_x) {
>     printf("found distinct x: %d\n", x);
>     last_x = x;
>   }
> }

That works too but ...

Temporary arrays are a problem from a memory allocation perspective:
we just don't like doing it at run time, and arrays are too big to
put into the contect struct.


> Q10: How does the interpretation of probe bodies and function bodies differ?
> -------
>
> From what I've been able to tell, functions get expanded into probe
> bodies, somewhat macro-like.

They could be, but actually they're translated to distinct blobs of C,
which are called/shared from multiple caller probes as needed.

> How do functions and macros differ? 

Macros are parse-time expansions (so they disappear by -p1); functions
are ... well like functions elsewhere.

> What can be done in probe bodies but not functions? There are clearly
> some differences since as my prior mail notes, functions resolve @cast
> and @var differently.

That's becuase reusable functions don't have a natural context to
resolve types/$variables within.  Probes do - the probe point itself.


> Q11: How to "fire" an "event" from a tapset?
> --------
> [...]
> Is using aliases the wrong approach? If you want users to be able to
> say "run this handler when event y occurs", and event Y may happen 6
> different ways with different relevant local variables etc each time,
> how do you do it?

You may be overthinking it: just say     probe y  { ... }
declaratively.  If you want to turn it on or off, use arming
conditionals or normal conditionals.


> Can a tapscript "fire" a synthetic probe? And put it in a context
> where it can access a process's globals etc?

Nope, we cannot do something as messy as gdb's inferior-function-calls.


> Q12: How to make a probe or probe alias show up or not show up in --monitor
> --------
> --monitor mode seems to have some kind of magic for deciding what
> probes it shows hits for.
> So far I've only seen it produce output for SDT markers, not function
> probes, function return probes, etc.

They should all show up, once they start getting hits.

> How does it find out when a probe is hit and count it? Is there any
> way to suppress a probe as a "hit" within the body, given that per Q1
> the probe conditional expression seems pretty limited.

When you ask "how ...." - you can gird your loins and look at the
output of "stap -p3 ....".  It'll show you exactly how. :-)

> I didn't find much at all about --monitor mode yet.

Yeah, it's not something we made a big fuss about.  [man stap].


> Q13: Easy way to handle enums?
> --------
> [...]
> enum MyProgramEnum {
>    XX,
>    YY,
>    ZZ
> };
>
> manually to
>
> %define MyProgramEnum_XX %( 0 %)
> %define MyProgramEnum_XX %( 1 %)
> %define MyProgramEnum_XX %( 2 %)

Have you seen   @const()?  It's basically a wrapper for embedded-C.
[man stap]

Some dwarf magic should let us find enum declarations in scope of a
probe, this sounds like a good RFE.


> Q14: Handling differences in target program version
> ------------
> For the sake of an example, say that PostgreSQL version 11 has an
> argument / global variable "foo", and in PostgreSQL 12 it's gone,
> replaced with "foo_defined" and "foo_value". Details don't matter,
> point is that one symbol vanished, two different ones replace it.
>
> Is there any way to handle these version differences by making probes
> conditional on target expressions/values? E.g. "use this probe if
> @var("pg_version_num") > 120000, otherwise this other probe" ?

A probe handler can use @defined() and macro wrappers to adapt to
the presence or absence of context variables.  See our old pal
[man stap] and its friends [man stapprobes].  Many tapsets use
it too.

PR13009 would let probe point conditionals include @defined() tests
(which normally resolve to literals early during translation).


> Q15: variable/dynamic probes
> ----------
>
> Say I want to probe a bunch of different binaries within some basedir
> I want to pass as a script argument (or better yet, resolve from the
> PATH of a target binary).
>
> It seems I can't just
>     alias pg.backend = process(@1 "/bin/myproc");

This works for me:

stap -e '
      probe pg.backend = process(@1 "/bin/myproc") {}
      probe pg.backend.function("foo") {}
' /directory/prefix


> but because string literal auto-concatentation doesn't happen for
> macro-expanded strings

If so, this is a bug.


> New user experience, docs suggestions
> ===============================
>
>
> Document stringifying numbers
> ----------
>
> Concatenation with . doesn't stringify numbers.

Yup, because "." forms an important signal to the type inferences
that its operands are strings.


> Document that you just use sprintf to do it, or you may want to use
> the "println" statement that automatically handles concatenation and
> stringification.

Sure.


> Array wildcards, iterate-with-return need to be documented in BG
> -----------
>
> I found systemtap infinitely easier to use once I stumbled across the
> array wildcard support. This would've been immensely useful early on
> and I think it should be highlighted in the beginners' guide:
> [...]

I think we're suffering to some extent because we have too much
documentation - and not all of it is updated when some new feature
gets added.  [man stap] and [man stapprobes] are the masters where
everything should be included; the more prosey pdfs are secondary
and we often seem to neglect them.


> Document that $1 ... $n and $# are usable in preprocessor
> ---------
>
> I tried to write code that used $# to decide whether $2, etc were
> valid and got very confused that stap kept spamming warnings.  [...]

OK.  There is also the tapset/argv.stp file that defines a boringly
C-like argc & argv[] global variable pair.  Hm, it lacks the markup
needed to plop it into the generated man pages along with other
[man -k tapset::].


> Document that probe definitions cannot reference variables
> -----------
>
> When getting started, it's surprising that you cannot
>
>     pgbasedir = "/path/to/postgres";
>     alias pg.backend = process($pgbasedir . "/bin/postgres")
> and it might be nice to cover what you can and cannot do in probe arguments.

Probe points must be literals, as they must be evaluated at translate
time.  Therefore they cannot vary.


> No process-local / thread-local vars
> ------------
> Any non-probe-scoped variable, whether "global" or "private", is
> global to the whole tapscript. There's AFAICS no way to instance such
> variables per-process or per-thread automatically. So the beginners
> guide should mention this and suggest the pattern:  [...]

See the "this.stp" tapset for syntactic sugar for "hidden array
indexed with tid()", like:

     probe tp_syscall.read {
         @this1 = buf_uaddr
     }
     probe tp_syscall.read.return {
         buf_uaddr = @this1
         printf("%s", user_string(buf_uaddr))
     }


> SDT probe data types and argument names:
> -----------
>
> SDT probes have arguments $arg1... $argn, and no data type information
> is preserved. That's exceedingly unfortunate as they're otherwise a
> super useful tool. Is it feasible to do anything like embed the param
> names and types in the probes ELF section or an extension section of
> it, for reading by stap?

Tough as from a C macro call given values, we can't get type names as
literals that we can just include into the .sdt elf note.  Will think
about it.


> String access: user_string, user_string_warn
> -------------
> [...]

> Target array access
> -------------
> It'd be nice if the docs mentioned how to access target array
> variables or array-valued struct members. It's fairly obvious for
> locals:
>
>     $targetvar[index];
>     $targetstruct->arraymember[index];
>
> For @var and @cast it works similarly, but will result in confusing
> failures to resolve the variable or type if you do it wrong. Parens
> are allowed and seem to be useful (needed?). E.g.
>     (@var("globalarray"))[idx]
>     (@var("globalstruct")->arraymember)[idx]

OK. 

> Deref'ing scalar as array produces confusing symbol not found error

This is a diagnostic quality bug.


> Typecasting
> -----------
>
> Same warning for @cast as for @var re functions, module paths.
>
> Warn that @cast to a pointer-hiding typedef will produce confusing
> results, strange faults, etc.
>
>
> Queue stats
> -------------
>
>
> tapset crosslinking
> ---------------
>
> It'd be nice to see more x-linking of important tapset info in the
> main docs and beginner docs, e.g. linking to



Dude, thanks a lot for all this info.  It's worth virtual gold to hear
from users with fresh eyes.  For those cases where some missing feature
or a bug was identified (as opposed to RTFManPage), it would be great to
get them into the queue (sourceware.org/bugzilla), and we can most
certainly take patches, even little ones like your preferred wordings
for the documentation nits.

If you'd like me to open bugzilla's for my impression of them, can do,
but doing it yourself would get you into the system and get the request
& sample-desire worded just right.


- FChE

Re: Newbie Notes

[Craig Ringer]

On Thu, 31 Oct 2019 at 04:10, Frank Ch. Eigler <fche@redhat.com> wrote:

>
>
> > Q1: Can I make a probe conditional per-pid in the "if" clause? What
> > *can* I do there?
>
> The [man stap] page covers probe conditionals this way:
>
>        Probes may be decorated with an arming condition, consisting of a
>        simple boolean expres‐sion on read-only global script variables.
>        While disarmed (inactive, condition evaluates to false), some
>        probe types reduce or eliminate their run-time overheads.  When
>        an arming condition evaluates to true, probes will be soon
>        re-armed, and their probe handlers will start getting called as
>        the events fire.  [...]
>
> The key is that these conditions are -arming conditions-.  They cannot
> possibly use contextual values such as pid(), because in order to even
> evaluate that condition, the probe would have to be armed & running!  In
> contrast, "boolean expressions on read-only global variables" may be
> evaluated anywhere: namely at the conclusion of all the probes that
> actually modify those variables.  Then those probes can enqueue an
> arm/disarm operation.
>

OK, that makes sense. I read over the manpage a few times, but I must've
missed that. My apologies.


> > Q2: Can I "return" or "exit" from a probe in a handler body or probe
> > alias handler body?
>
> "next", as in awk.  This too is in [man stap].
>

Ah! Thanks. Right there in "STATEMENTS" too.

Also in https://sourceware.org/systemtap/langref/6_Statement_types.html .

Chalk one up to me being blind. My apologies. It might be handy to mention
the keyword "return" or "exit" there for search convenience for similarly
reading-ability-impaired folks.


> Unused variables defined in aliases should not generate any warnings; or
> maybe you're using "stap -u" ?
>

Nope, I noticed it with regular "stap". I'll re-check and see if I can tell
what was going on and come back with details...

OK. Take this script, which will work on anything you have lying around
with simple adaptations:

    probe pg = process("postgres")
    {
        v1 = pid();
        v2 = execname();
        v3 = "availble for future callers";
    }

    probe pg.function("PostgresMain")
    {
        printf("my pid is %d and name is %s\n", v1, v2);
    }

    probe pg.function("PostmasterMain")
    {
        printf("I don't use those variables\n");
    }

Since the variables defined here are in the prelude handler of the alias I
wouldn't expect warnings for them, but on either stap 4.1 or
4.2-git-cadeeaa25 I get:

    sudo PATH=/usr/pgsql-11/bin:$PATH ./stap -v warnings.stp

    Pass 1: parsed user script and 476 library scripts using
201060virt/78756res/9736shr/69012data kb, in 150usr/30sys/176real ms.
    WARNING: Eliding assignment to 'v3': operator '=' at warnings.stp:5:12
     source:         v3 = "availble for future callers";
                        ^
    WARNING: Eliding assignment to 'v1': operator '=' at :3:12
     source:         v1 = pid();
                        ^
    WARNING: Eliding assignment to 'v2': operator '=' at :4:12
     source:         v2 = execname();
                        ^
    WARNING: Eliding assignment to 'v3': operator '=' at :5:12
     source:         v3 = "availble for future callers";
                        ^
    WARNING: Eliding side-effect-free expression : identifier 'v3' at :5:9
     source:         v3 = "availble for future callers";
                     ^
    WARNING: Eliding side-effect-free expression : identifier 'v1' at :3:9
     source:         v1 = pid();
                     ^
    WARNING: Eliding side-effect-free expression : identifier 'v2' at :4:9
     source:         v2 = execname();
                     ^
    WARNING: Eliding side-effect-free expression : identifier 'v3' at :5:9
     source:         v3 = "availble for future callers";
                     ^

I'm inclined to suspect a severe case of PEBKAC is at work here, but it
wouldn't be the first time I've found that when I speak up about something
that confuses me it's confused others in the past too.


> > Q4: tapscript backtraces from runtime faults, errors, etc, macro
> > expanded-from for errors at runtime?
> > --------------
> >
> > Say I have some macro @MY_MACRO(x) or some function my_function(x).
> > I do something in the macro that raises an error() or fault.
> > The error from stap only reports the line number of the macro or
> function.
>
> Aha.  Hmm, this has come up before, but I don't think we recorded it
> properly as an RFE (request-for-enhancement) BZ.  It shouldn't be too
> hard, given that we do track statement-by-statement where we are, in the
> context->last_stmt variable.  We would have to make that nestable, by
> storing it inside the context->locals[LEVEL] struct instead.  It's a bit
> of work but not that much.  Might you be interested in giving
> implementing it a try?
>

Definitely interested. I'll have to check with work about getting the time
since I'm currently rather short of outside-work coding time, but I think
stap looks like a tool we could really benefit from.

(I'm also getting into full-stack tracing with Jaeger and now OpenCensus,
but that's much more high level and heavyweight, so both have a role).


> @define @length1(a,v) %( @v = 0; foreach (_x in @a) { @v ++ } %)
> @define @length2(a,v) %( @v = 0; foreach (_x,_y in @a) { @v ++ } %)
>

Thanks. Was mostly checking if I was missing something, a-la "next".


> > Q6: is there any way to get the executable-path of the current pid()
> > in a probe Just that.
>
> task_execname(task_current())   [man tapset::task]
>

Since posting, I found function::execname in tapset::context .

It's not totally clear to me how that differs from
using task_execname(task_current()), if at all.

It'd be good to mention execname() in the beginners' guide alongside pid()
and to see-also between those funcs.


>
> > Q7: Is there a string-concatenation operator that works at parse-time
> > with macro-expansion?
> > Say I want to use
> >
> >    @define MY_PROG_BASEPATH %{ @1 %}
> >    @define MY_PROG(prog) %{ @MY_PROG_BASEPATH @prog %}
>
> Same rules as in C: adjacent string literals concatenate during parse.
>


The issue is specifically within @var and @cast expansion. It's done early,
and doesn't appear to benefit from implicit string concatenation at all,
macro-expanded or otherwise. Per my other post.

This works:

  @define PGBIN %( @1 "/bin/postgres" %)
  probe begin { printf("%s\n", @PGBIN); }

so does this:

    @define PGBIN %( @1 "/bin/postgres" %)
    probe process(@PGBIN).function("PostgresMain") { printf("%s\n",
@PGBIN); }

but not this (where $1 = /usr/pgsql-11/ ):

        @define PGBIN %( @1 "bin/postgres" %)
        function get_pgver:long() {
            return @var("server_version_num@guc.c", @PGBIN);
        }
        probe process(@PGBIN).function("PostgresMain") {
            printf("%s", @PGBIN, get_pgver());
        }

producing:

    parse error: expected ')'
        saw: string 'bin/postgres' at concat5.stp:1:29
         source:         @define PGBIN %( @1  ... bin/postgres ...
"bin/postgres" %)
                              ^
        in expansion of macro: operator '@PGBIN' at concat5.stp:3:53
         source:             return @var("server_version_num@guc.c",
@PGBIN);
                                     ^

    1 parse error.
    Pass 1: parse failed.  [man error::pass1]

however if I substitute the @var line with

            return @var("server_version_num@guc.c",
"/usr/pgsql-11/bin/postgres");

it works.

I haven't worked out exactly why yet, but it looks likely to be a matter of
order-of-processing. I wonder if @var @cast etc need to be processed in a
pass *after* user defined macros.


> We don't have variadic macros, and variadic functions don't quite
> propagate string literals well enough for this to work:
>
> function warn(msg,fmt,var1) { warn(sprint("%s " fmt, msg, var1)) }
> function warn(msg,fmt,var1,var2) { warn(sprint("%s " fmt, msg, var1,
> var2)) }
>

No worries, just figured it was worth checking.


> > Q9: Can I iterate over a distinct-values slice of an array?
> > -------
> > Say I have an array arr[x,y,z]. I want to iterate over all distinct
> > "x" without repeats.
> > There are no local associative array vars.
>
> Yeah, it'd require about the same amount of temporary storage as if you
> spelled out the pair of operations consecutively:
>
>     foreach ([x,y,z] in array)  array2[x] = 2
>     foreach ([unique] in array2) /* bob is unique */ ;
>

Right - but the docs say global array access requires locking and there are
no local arrays. Isn't that a performance/overhead concern?

It's not a big deal as the wonderful sorted-array-traversal operations let
you just notice when there's a distinct key being visited. Just a touch
verbose.

Temporary arrays are a problem from a memory allocation perspective:
> we just don't like doing it at run time, and arrays are too big to
> put into the contect struct.
>

Makes sense, especially when there's all that concern about contiguous
memory, fragmentation etc when working in the kernel.


>  Q10: How does the interpretation of probe bodies and function bodies
> differ?
> > -------
> >
> > From what I've been able to tell, functions get expanded into probe
> > bodies, somewhat macro-like.
>
> They could be, but actually they're translated to distinct blobs of C,
> which are called/shared from multiple caller probes as needed.
>

Huh. I assumed they were expanded because it looked like when I called the
same function from multiple probes, any warnings raised by the function got
emitted repeatedly. I may be mistaken.

It would be nice to document this alongside @cast and @var as it's very
confusing as a new user to move something into a function and have its
behaviour change suddenly.

Come to think of it, wouldn't it be pretty easy to use the focus_on_module
stuff to let functions be annotated with a default target module, or to
define blocks with target modules? Think

    function get_application_name:string() default_module("postgres")
    {
        return @var("application_name@guc.c");
    }

or

    function get_application_name:string()
    {
        @default_module("postgres") {
            return @var("application_name@guc.c");
        }
    }

Pointless in the simple case above, but not in more complex deployments.

OTOH, the lead time for anything like this to reach production systems is
so long it's questionable whether it's worth bothering with as any scripts
will need workarounds in the meantime.



> > What can be done in probe bodies but not functions? There are clearly
> > some differences since as my prior mail notes, functions resolve @cast
> > and @var differently.
>
> That's becuase reusable functions don't have a natural context to
> resolve types/$variables within.  Probes do - the probe point itself.
>

Makes sense given what you said above; my assumption about functions was
incorrect. And there'd be no way to infer the context from the call site
without specializing the function into callsite-specific variants, which
would be a nice feature to have but clearly not one that's present now.


> > Q11: How to "fire" an "event" from a tapset?
> > --------
> > [...]
> > Is using aliases the wrong approach? If you want users to be able to
> > say "run this handler when event y occurs", and event Y may happen 6
> > different ways with different relevant local variables etc each time,
> > how do you do it?
>
> You may be overthinking it: just say     probe y  { ... }
> declaratively.  If you want to turn it on or off, use arming
> conditionals or normal conditionals.
>

Won't work for me because I'm working in a complex multi-process context
and I can't arm/disarm probes per-process.

But I can just have my alias handler do

   if (foo) { next; }

which is fine.


> > Can a tapscript "fire" a synthetic probe? And put it in a context
> > where it can access a process's globals etc?
>
> Nope, we cannot do something as messy as gdb's inferior-function-calls.
>

Yeah. No ptrace(), non-invasive, process is actively executing, working in
kernel space, all that.

I was thinking more of something like you have with associative arrays,
synthetic probes on timer() events etc. Register an event (internally a
global) at compile time. Fire it to trigger the handler.

But it's not really necessary if probe aliases aren't meant to warn on
unused vars and I can "next" from a probe alias handler. I'll confirm it
works as expected in my test case...

> Q12: How to make a probe or probe alias show up or not show up in
> --monitor
> > --------
> > --monitor mode seems to have some kind of magic for deciding what
> > probes it shows hits for.
> > So far I've only seen it produce output for SDT markers, not function
> > probes, function return probes, etc.
>
> They should all show up, once they start getting hits.
>

Weird, they don't here. Will install latest stap and look into it; while
I've studied the latest git sources I've still been testing against

     Systemtap translator/driver (version 4.1/0.177/0.176, rpm 4.1-2.fc30)





> > How does it find out when a probe is hit and count it? Is there any
> > way to suppress a probe as a "hit" within the body, given that per Q1
> > the probe conditional expression seems pretty limited.
>
> When you ask "how ...." - you can gird your loins and look at the
> output of "stap -p3 ....".  It'll show you exactly how. :-)
>

Thanks. I've spent a while browsing that output already. I guess I may've
been unclear; I'm talking about how --monitor decides what is / isn't a
hit. Does using "next" in a probe alias body suppress it? etc.

As a heavy user of perf's "perf top", stap's --monitor is of great interest
to me.

I hadn't had time to read the monitor code before I posted this. I had a
look now and AFAICS it's injecting code into the tapscript that probes
procfs "monitor_status" to generate some json summary output about globals,
probes, etc from a probe done in ().

It calls __private___monitor_data_function_probes for each probe, which is
defined in gen_monitor_data() in a bit of a wall of inline code as
cstrings. That's embedded C (?) that looks up struct stap_probes which is
defined by translate_pass() to create the macro STAP_PROBE_INIT(...) .

I got a bit lost in the C++ around struct probe and struct statement at
that point, but some guesswork took me to common_probe_entryfn_prologue and
STP_TIMING which uses skipped_count(), but that seems to be session-level
not per-probe. STP_ALIBI seems to generate a per-probe atomic_inc(...) but
I didn't track that back to where it's defined or why.



> > I didn't find much at all about --monitor mode yet.
>
> Yeah, it's not something we made a big fuss about.  [man stap].
>
>
> > Q13: Easy way to handle enums?
> > --------
> > [...]
> > enum MyProgramEnum {
> >    XX,
> >    YY,
> >    ZZ
> > };
> >
> > manually to
> >
> > %define MyProgramEnum_XX %( 0 %)
> > %define MyProgramEnum_XX %( 1 %)
> > %define MyProgramEnum_XX %( 2 %)
>
> Have you seen   @const()?  It's basically a wrapper for embedded-C.
>

Yeah, I did see that, but you can't use embedded C normal in user tap
scripts, only tapsets or guru mode, and as I mentioned I'm more than a tad
nervous about trying to include/parse complex and portable headers like
those in postgres. Plus that way you need the headers on hand.

Not a biggie, it'd just be convenient to be able to copy the enums.


> Some dwarf magic should let us find enum declarations in scope of a
> probe, this sounds like a good RFE.
>

Huh, good point. I was only thinking parse/compile time, didn't occur to me
it could be done at runtime. @enum(...) and if using -ggdb3 maybe
even @macro(...) for simple integers. But I can't volunteer for that one,
so it's likely to be an idle-for-a-long-time RFE unless I can get some
allocated time for stap.


> > Q14: Handling differences in target program version
> > ------------
> > For the sake of an example, say that PostgreSQL version 11 has an
> > argument / global variable "foo", and in PostgreSQL 12 it's gone,
> > replaced with "foo_defined" and "foo_value". Details don't matter,
> > point is that one symbol vanished, two different ones replace it.
> >
> > Is there any way to handle these version differences by making probes
> > conditional on target expressions/values? E.g. "use this probe if
> > @var("pg_version_num") > 120000, otherwise this other probe" ?
>
> A probe handler can use @defined() and macro wrappers to adapt to
> the presence or absence of context variables.  See our old pal
> [man stap] and its friends [man stapprobes].  Many tapsets use
> it too.
>
> PR13009 would let probe point conditionals include @defined() tests
> (which normally resolve to literals early during translation).
>
>
> > Q15: variable/dynamic probes
> > ----------
> >
> > Say I want to probe a bunch of different binaries within some basedir
> > I want to pass as a script argument (or better yet, resolve from the
> > PATH of a target binary).
> >
> > It seems I can't just
> >     alias pg.backend = process(@1 "/bin/myproc");
>
> This works for me:
>
> stap -e '
>       probe pg.backend = process(@1 "/bin/myproc") {}
>       probe pg.backend.function("foo") {}
> ' /directory/prefix
>
>
> > but because string literal auto-concatentation doesn't happen for
> > macro-expanded strings
>
> If so, this is a bug.
>
>
> > New user experience, docs suggestions
> > ===============================
> >
> >
> > Document stringifying numbers
> > ----------
> >
> > Concatenation with . doesn't stringify numbers.
>
> Yup, because "." forms an important signal to the type inferences
> that its operands are strings.
>

Which makes sense. There are many extremely useful hints in systemtap
already and this might be a candidate for another though:

    $ sudo stap -e 'probe begin { a = 1; b = "b" . a; printf("b is %s\n",
b); }'
    semantic error: type mismatch (string): identifier 'a' at <input>:1:32
            source: probe begin { a = 1; b = "b" . a; printf("b is %s\n",
b); }
                                                   ^

    semantic error: type was first inferred here (long): identifier 'a' at
:1:15
            source: probe begin { a = 1; b = "b" . a; printf("b is %s\n",
b); }

for operator(.) like "operator '.' takes string operands, do you need
sprint() or sprintf()? see [function::sprint], [function::sprintf]"

?

Unsure how simple it is to determine the operator (.) is the cause of the
type mismatch though and frankly it's probably easier to just add a note
alongside the . operator in the docs pointing to sprint() and sprintf() for
coercing integers to strings.

A note on the print/sprint entry in the manpage that mentions the keyword
"integer" wouldn't hurt either. I'll see if I can put together a docs PR
soon.



> > Array wildcards, iterate-with-return need to be documented in BG
> > -----------
> >
> > I found systemtap infinitely easier to use once I stumbled across the
> > array wildcard support. This would've been immensely useful early on
> > and I think it should be highlighted in the beginners' guide:
> > [...]
>
> I think we're suffering to some extent because we have too much
> documentation - and not all of it is updated when some new feature
> gets added.


Right, I get that. You've pointed out places where things were documented
and I didn't find them because I was doing targeted searches and reading
key sections, having not read it cover-to-cover yet.

I'm prone to writing too much documentation that people don't read myself.


> > Document that $1 ... $n and $# are usable in preprocessor
> > ---------
> >
> > I tried to write code that used $# to decide whether $2, etc were
> > valid and got very confused that stap kept spamming warnings.  [...]
>
> OK.  There is also the tapset/argv.stp file that defines a boringly
> C-like argc & argv[] global variable pair.
>

Huh, handy. That's good to know :) . TBH I probably would've assumed the
argv tapset was for target arguments anyway had I seen it ; it's worth
x-ref'ing where $1 @1 etc are doc'd in man stap.


> > Document that probe definitions cannot reference variables
> > -----------
> >
> > When getting started, it's surprising that you cannot
> >
> >     pgbasedir = "/path/to/postgres";
> >     alias pg.backend = process($pgbasedir . "/bin/postgres")
> > and it might be nice to cover what you can and cannot do in probe
> arguments.
>
> Probe points must be literals, as they must be evaluated at translate
> time.  Therefore they cannot vary.
>

Yeah, I get that now :) . This is one of those "confused me at the
beginning" things.

I didn't know about macros yet.

> No process-local / thread-local vars
> > ------------
> > Any non-probe-scoped variable, whether "global" or "private", is
> > global to the whole tapscript. There's AFAICS no way to instance such
> > variables per-process or per-thread automatically. So the beginners
> > guide should mention this and suggest the pattern:  [...]
>
> See the "this.stp" tapset for syntactic sugar for "hidden array
> indexed with tid()", like:
>
>      probe tp_syscall.read {
>          @this1 = buf_uaddr
>      }
>      probe tp_syscall.read.return {
>          buf_uaddr = @this1
>          printf("%s", user_string(buf_uaddr))
>      }
>

Yeah, I found that, but there's no equivalent for pid() ....

ugh, and I'm so stupid it hurts. For non-threaded processes pid()
corresponds 1:! to tid() . Sigh.


> > SDT probe data types and argument names:
> > -----------
> >
> > SDT probes have arguments $arg1... $argn, and no data type information
> > is preserved. That's exceedingly unfortunate as they're otherwise a
> > super useful tool. Is it feasible to do anything like embed the param
> > names and types in the probes ELF section or an extension section of
> > it, for reading by stap?
>
> Tough as from a C macro call given values, we can't get type names as
> literals that we can just include into the .sdt elf note.  Will think
> about it.
>

Right, not saying it's easy ;)

DWARF with -ggdb3 has the macro definition, but probably not in a form
convenient to map through the layers of expansion back to the original
argument types. I think it's just the raw definition.

I was thinking about it mostly for SDTs defined by probes.d, which already
syntatically accepts c-alike probe definitions, it just ignores the
argument names and types completely AFAICS. In this case the dtrace script
could generate a second macro alongside the DTRACE_PROBE2 or whatever like
a

    STAP_PROBE_ARG_NAMES2("somearg", "someotherarg");
    STAP_PROBE_ARG_TYPES2("const char *", "uint32_t");

which can presumably be inserted into a custom ELF section in much the same
manner as the probe points themselves are generated.

Whether it's useful to do so is another matter, though, as it'd require
other tools to agree upon and adopt the scheme. Also, this way the type
information would only be strings, which limits options for type checking
and for doing anything useful with it in the face of typedefs etc.

Dude, thanks a lot for all this info.  It's worth virtual gold to hear
> from users with fresh eyes.


I hoped so. I know it can also be frustrating because to you it's "just
right there dammit!" - but new users face information overload, are trying
to adapt to new terminology and structures, and are usually under time
pressure. So it's not just whether it's documented, but where and how.

I wish I could do better at that myself. Sometimes I feel like the only
user of some of the docs I write is ... myself.


> For those cases where some missing feature
> or a bug was identified (as opposed to RTFManPage), it would be great to
> get them into the queue (sourceware.org/bugzilla), and we can most
> certainly take patches, even little ones like your preferred wordings
> for the documentation nits.
>
> If you'd like me to open bugzilla's for my impression of them, can do,
> but doing it yourself would get you into the system and get the request
> & sample-desire worded just right.
>

Can do. I'll try to prep the odd patch as well.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Newbie Notes

[Frank Ch. Eigler]

Hi -


> OK. Take this script, which will work on anything you have lying around
> with simple adaptations:
> [...]

Ah.  Yes, the unused-variable warnings are suppressed, but only for
script located in a tapset.  If you put that first alias definition
into a separate directory/file, and run with stap -I$dir, then you
will not see those warnings.


> The issue is specifically within @var and @cast expansion. It's done early,
> and doesn't appear to benefit from implicit string concatenation at all,
> macro-expanded or otherwise. Per my other post.  [...]
> but not this (where $1 = /usr/pgsql-11/ ):
> 
>         @define PGBIN %( @1 "bin/postgres" %)
>         function get_pgver:long() {
>             return @var("server_version_num@guc.c", @PGBIN);
>         }
>         probe process(@PGBIN).function("PostgresMain") {
>             printf("%s", @PGBIN, get_pgver());
>         }

OK, working on this bug.


> [...] I'm talking about how --monitor decides what is / isn't a
> hit. Does using "next" in a probe alias body suppress it? etc.

It should count every distinct probe whose handler starts executing,
so definitely include those that run through to a 'next'.

> As a heavy user of perf's "perf top", stap's --monitor is of great interest
> to me.

By the way, see also   stap --example eventcount.stp 

> [...]


- FChE

Re: Newbie Notes

[Craig Ringer]

On Fri, 8 Nov 2019 at 02:52, Frank Ch. Eigler <fche@redhat.com> wrote:

> Hi -
>
>
> > OK. Take this script, which will work on anything you have lying around
> > with simple adaptations:
> > [...]
>
> Ah.  Yes, the unused-variable warnings are suppressed, but only for
> script located in a tapset.  If you put that first alias definition
> into a separate directory/file, and run with stap -I$dir, then you
> will not see those warnings.
>

Aha. Makes sense.

I can work around the PATH issues with @var etc by generating a .tapm
script with the paths templated in, then use that in a tapset that defines
the probe aliases etc.

That'll work well enough that I think I can prepare a usable tapset for
PostgreSQL.

(Ideally stap would only warn if a variable was unused across all
references to an alias. So if any probe body used the alias-defined
variable it wouldn't warn. But from my initial reading of the relevant code
that doesn't look simple.)

> The issue is specifically within @var and @cast expansion. It's done
> early,
> > and doesn't appear to benefit from implicit string concatenation at all,
> > macro-expanded or otherwise. Per my other post.  [...]
> > but not this (where $1 = /usr/pgsql-11/ ):
> >
> >         @define PGBIN %( @1 "bin/postgres" %)
> >         function get_pgver:long() {
> >             return @var("server_version_num@guc.c", @PGBIN);
> >         }
> >         probe process(@PGBIN).function("PostgresMain") {
> >             printf("%s", @PGBIN, get_pgver());
> >         }
>
> OK, working on this bug.
>

Cool. I dug through a fair bit of code without figuring it out, sorry :S

Want me to open a github issue?

I've started working around this in two ways:

1. If possible use @cast and @var only in probe bodies, including aliases,
chaining aliases etc where required, bypassing the issue with module
context in functions entirely; or
2. Where that's not possible, generating a stub .stpm that defines the
required full executable paths from a wrapper script, Makefile, etc, then
using those macros in the main tapset.

I'll be stuck being backward compatible for a while so I had to work around
it anyway...


> > [...] I'm talking about how --monitor decides what is / isn't a
> > hit. Does using "next" in a probe alias body suppress it? etc.
>
> It should count every distinct probe whose handler starts executing,
> so definitely include those that run through to a 'next'.
>

Cool.

I guess in that case I'd ideally like a way to suppress a probe hit, so the
probe body can say "nah, actually this isn't what I'm looking for". But
it's hardly important, and if I need it I can write it, so consider it a
vague handwavey idea only.


> > As a heavy user of perf's "perf top", stap's --monitor is of great
> interest
> > to me.
>
> By the way, see also   stap --example eventcount.stp
>
>
Thanks.


-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Newbie Notes

[Frank Ch. Eigler]

Hi -

> Aha. Makes sense.
> 
> I can work around the PATH issues with @var etc by generating a .tapm
> script with the paths templated in, then use that in a tapset that defines
> the probe aliases etc.
> 
> That'll work well enough that I think I can prepare a usable tapset for
> PostgreSQL.

And if that tapset is in a tapset directory, then the unused var
warnings won't be an issue.


> > OK, working on this bug.
> 
> Cool. I dug through a fair bit of code without figuring it out, sorry :S
> Want me to open a github issue?

Nah, already fixed it last night.  (We use sourceware.org/bugzilla.)

> [...]
> I'll be stuck being backward compatible for a while so I had to work around
> it anyway...

Fair enough.  When we release a new version, we do plop it onto as
many old platform versions as we can.  So fedora30 etc. should get it,
within a few weeks.


- FChE

Re: Newbie Notes

[Craig Ringer]

On Fri, Nov 8, 2019, 20:03 Frank Ch. Eigler <fche@redhat.com> wrote:

> Hi -
>
> > Aha. Makes sense.
> >
> > I can work around the PATH issues with @var etc by generating a .tapm
> > script with the paths templated in, then use that in a tapset that
> defines
> > the probe aliases etc.
> >
> > That'll work well enough that I think I can prepare a usable tapset for
> > PostgreSQL.
>
> And if that tapset is in a tapset directory, then the unused var
> warnings won't be an issue.
>

Yep. When I clean up my probes into a tapset I'll post about it here.


>
> > > OK, working on this bug.
> >
> > Cool. I dug through a fair bit of code without figuring it out, sorry :S
> > Want me to open a github issue?
>
> Nah, already fixed it last night.  (We use sourceware.org/bugzilla.)
>

Wow, thanks. Pulling to have a look.



> > [...]
> > I'll be stuck being backward compatible for a while so I had to work
> around
> > it anyway...
>
> Fair enough.  When we release a new version, we do plop it onto as
> many old platform versions as we can.  So fedora30 etc. should get it,
> within a few weeks.
>

Very nice!

My work actually runs our own package distribution system now too, so if we
needed to we could even build and ship systemtap.

Which reminds me, I need to tweak our packages to force the installation of
debuginfo.


>
> - FChE
>
>