Path resolution inconsistency between process("binary") and @var("var@cu","module")

Path resolution inconsistency between process("binary") and @var("var@cu","module")

[Craig Ringer]

Hi all

TL;DR: @var("...","module") doesn't resolve "module" against PATH like
process("execname") does; confusion ensures. SCCE included, fix
proposed, comments requested.

----

I've recently discovered how amazing and powerful SystemTap is, after
having it on my backburner to learn for years while struggling to
achieve anything with linux-perf.

I've been keeping notes on some suggestions to make for the
documentation which I'll send separately. But first I've found (and
worked around) an issue with user-space probing that I'd like to
patch, but I'm looking for confirmation and advice first.

The gist is that the PATH-resolution done by "process(...)" probes is
not consistent with how @var(...) and @cast(...) resolve modules
within stap functions called by probes. This causes stap to report
very confusing failures to resolve symbols that are obviously right
there as far as the user is concerned.

The working cases are, for some probe on a function/statement in a
userspace executable:

- @var("sym@CU") directly in a probe; doesn't require module argument at all
- @var("sym@CU", "/full/path/to/executable") in a function() called by
a probe: finds var

The failing cases are, all when in a function() called by a probe of a
userspace executable:

- @var("sym@CU") in the function: reports missing symbol, tries to
resolve in "kernel"
- @var("sym@CU","executable") in the function: also tries to resolve in kernel

The probe type doesn't seem to matter that much:
process("executable"), process("/path/to/executable"), process() and
target mode, process(pid), etc etc all behave the same. @cast has
similar issues to @var.

SCCE with cases shown:

program to probe:

     /* path_vs_atvar.c */
     #include <stdlib.h>
     static int global_variable = 0;
     int main(int argc, char *argv[])
     {
         global_variable = argc > 1 ? atoi(argv[1]) : 0;
         return global_variable;
     }
     /* end path_vs_atvar.c */

compilation:

    gcc -ggdb3 -Wall path_vs_atvar.c -o path_vs_atvar

tapscript:

     function get_global()
     {
         // Doesn't work, preprocesses to @var(...,"kernel") [1]
         //return @var("global_variable@path_vs_atvar.c");

         // Doesn't work, module argument treated as kernel [2]
         //return @var("global_variable@path_vs_atvar.c","path_vs_atvar");

         // Fully qualified path as literal string: works,
cumbersome/impractical [3]
         //return
@var("global_variable@path_vs_atvar.c","/home/craig/projects/scrapcode/systemtap/path-vs-atvar/path_vs_atvar");

         // Path as script-arg: works, though also cumbersome as user
must resolve
         // and specify path as absolute path which defeats the point
of path-expansion
         // in process probes, doesn't work well with -c, etc. [4]
         return @var("global_variable@path_vs_atvar.c",@1);
     }

     probe process.function("main").return
     {
         // Works: resolves variable in correct module for any
suitable probe type like:
         //
         //     process("path_vs_atvar")
         //     process("/path/to/path_vs_atvar")
         //     process($1)  with path arg
         //     process() using -c mode
         //
         // etc. [0]
         //
         //printf("global is %d\n", @var("global_variable@path_vs_atvar.c"));

         // Wheras when we indirect via a function, results vary
         printf("global is %d\n", get_global());
     }


To invoke, uncomment the [case] you want to try and run:

    sudo stap path_vs_atvar.stp -c ./path_vs_atvar


for case [1] the result is:


     probe main@/home/craig/projects/scrapcode/systemtap/path-vs-atvar/path_vs_atvar.c:3
process=/home/craig/projects/scrapcode/systemtap/path-vs-atvar/path_vs_atvar
reloc=.absolute pc=0x401126
     semantic error: unresolved @var() expression: operator '@var' at
path_vs_atvar.stp:4:12
        thrown from: elaborate.cxx:6621
             source:     return @var("global_variable@path_vs_atvar.c");
                                ^

     Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals
using 262184virt/143804res/13596shr/129868data kb, in
60usr/120sys/187real ms.
     Missing separate debuginfos, use: debuginfo-install
kernel-core-5.3.5-200.fc30.x86_64


You'd expect case [1] above, with no module-spec, to resolve the @var
in the context the function is called in, but it doesn't. It's trying
to use the "kernel" module per -vvv:

    return @var("global_variable@probe_vs_atvar.c", "kernel")
    semantic error: unresolved @var() expression: operator '@var' at
path_vs_atvar.stp
       thrown from: elaborate.cxx:6621
            source:     return @var("global_variable@probe_vs_atvar.c");


so that's weird. The global is clearly there and in the specified CU:

    $ gdb -q -ex "info variables global_variable" -ex quit ./path_vs_atvar
    Reading symbols from ./path_vs_atvar...
    All variables matching regular expression "global_variable":

    File path_vs_atvar.c:
    2: static int global_variable;
    $

But if you change it to explicitly reference the module
"probe_vs_atvar" per [2] the outcome is the same, even though we
*know* the module "path_vs_atvar" must exist and be accessible since
we're being invoked from a .process probe on it.

After all, it works when directly within the probe body [0].

If you use an absolute path to the module then the @var works within
the function [4], e.g.

    @var("global_variable@path_vs_atvar.c","/path/to/path_vs_atvar");

but that's exceedingly impractical.

The most tolerable workaround I found so far was to use a commandline
argument to specify the full path to the executable. This makes
process PATH resolution completely useless, but at least it works,
like in case [5] in the SCCE.

As I have quite a large set of executables to probe I tried to
simplify use by using a macro to expand sub-paths from a basedir, but
that doesn't work either. Implicit string concatenation doesn't get
repeated after @MACRO expansion so @var("sym@CU", @MYBASEDIR
"bin/foo")  won't work. I didn't find any language support for
preprocessing-time explicit string concatenation.

So here I am.

I did some code digging. It's my first time looking at the sytemtap
code and I've only used it for a week, so please forgive
errors/oversights, but AFAICS the important code is
dwarf_atvar_expanding_visitor(...) and parse_atvar_op(...).

I haven't quite worked out why the case in [0] works but I'm assuming
it's something to do with focus_on_module().

For the function cases, I think this is partly due to the
is_user_module() call in setupdwfl.cxx which assumes that userspace
modules must always be absolute paths.

  bool
  is_user_module(const std::string &m)
  {
    return m[0] == '/' && m.rfind(".ko", m.length() - 1) != m.length() - 3;
  }

That strikes me as an awfully bogus assumption; it should be possible
to specify a userspace module by bare-name in @var, @cast, etc not
just a kernel module.

But taking a step back: systemtap appears to expand functions once per
invocation in a probe anyway, so why doesn't the function benefit from
the focus_on_module() of the probe and work like case [0] without any
specification of module name at all?

Phew! Ideas?



-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Path resolution inconsistency between process("binary") and @var("var@cu","module")

[Craig Ringer]

Further

> But taking a step back: systemtap appears to expand functions once per
> invocation in a probe anyway, so why doesn't the function benefit from
> the focus_on_module() of the probe and work like case [0] without any
> specification of module name at all?
>
> Phew! Ideas?


Further note on this. I since tried the auto_path feature by putting
my library tapset in

    tapsets/PATH/my/path/to/postgres

and used

    stap -I `pwd`/tapsets/ test.stp

but had exactly the same issues. A look at the code suggests that the
auto_path feature, like the PATH search etc, is only effective for
probes, not for @var and @cast .



-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 2ndQuadrant - PostgreSQL Solutions for the Enterprise

Re: Path resolution inconsistency between process("binary") and @var("var@cu","module")

[Frank Ch. Eigler]

craig wrote:

> [...]
> The working cases are, for some probe on a function/statement in a
> userspace executable:
>
> - @var("sym@CU") directly in a probe; doesn't require module argument at all
> - @var("sym@CU", "/full/path/to/executable") in a function() called by
> a probe: finds var
>
> The failing cases are, all when in a function() called by a probe of a
> userspace executable:
>
> - @var("sym@CU") in the function: reports missing symbol, tries to
> resolve in "kernel"
> - @var("sym@CU","executable") in the function: also tries to resolve in kernel

The reason for this is that a function does NOT have a probe context
relative to which it can do dwarf symbol resolution.  It does not
operate on some sort of auto-inherited context from its caller probe
handler(s) and/or function(s).  So, disambiguation of kernel-space
(including module names!) vs.  user-space (executables) is left up to a
simple naming convention in the function you found.  Having a $PATH
based search would improve convenience at the cost of reintroducing
ambiguity.  Not sure whether that's worth it, are you sure it is?


- FChE

Re: Path resolution inconsistency between process("binary") and @var("var@cu","module")

[Craig Ringer]

On Thu, Oct 31, 2019, 06:15 Frank Ch. Eigler <fche@redhat.com> wrote:

>
> craig wrote:
>
> > [...]
> > The working cases are, for some probe on a function/statement in a
> > userspace executable:
> >
> > - @var("sym@CU") directly in a probe; doesn't require module argument
> at all
> > - @var("sym@CU", "/full/path/to/executable") in a function() called by
> > a probe: finds var
> >
> > The failing cases are, all when in a function() called by a probe of a
> > userspace executable:
> >
> > - @var("sym@CU") in the function: reports missing symbol, tries to
> > resolve in "kernel"
> > - @var("sym@CU","executable") in the function: also tries to resolve in
> kernel
>
> The reason for this is that a function does NOT have a probe context
> relative to which it can do dwarf symbol resolution.  It does not
> operate on some sort of auto-inherited context from its caller probe
> handler(s) and/or function(s).  So, disambiguation of kernel-space
> (including module names!) vs.  user-space (executables) is left up to a
> simple naming convention in the function you found.  Having a $PATH
> based search would improve convenience at the cost of reintroducing
> ambiguity.  Not sure whether that's worth it, are you sure it is?
>

Makes sense.

Pretty critically reduces the utility of functions though.

@uvar(...) and @ucast(...) like the convention elsewhere would make sense
for user vs kernel ambiguity. Or an extra optional argument specifying
resolution.

PATH resolution should be stable within a tap run so there isn't much
concern about differing resolutions in differing places.

I reckon I could implement that.