From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 49194 invoked by alias); 28 Oct 2019 05:53:05 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 49184 invoked by uid 89); 28 Oct 2019 05:53:04 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-1.8 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=ham version=3.3.1 spammy=Fully, workaround, forgive, literal X-HELO: mail-lf1-f52.google.com Received: from mail-lf1-f52.google.com (HELO mail-lf1-f52.google.com) (209.85.167.52) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Mon, 28 Oct 2019 05:53:02 +0000 Received: by mail-lf1-f52.google.com with SMTP id j14so1883410lfb.8 for ; Sun, 27 Oct 2019 22:53:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=qNRCiUNapx6JhtaBBFlRA8kCQPRIFTSFhrMkFFh31io=; b=kugx9/X7KPsPbTqInN3bmCBpWVnE/kM61WpmUdZ99QAa9Dcx1+1ND+hqTlm5DTVjnq BPJZpPh94yLc/mVPiD3SevBF1xjMHqeSGGy/AnkYSor+H3amZ4Txd0H+avtWovqGQSD+ oyYUPGIaql2ca00ABUn5eqOGVKf0iKrSrUb3BmJduZWzXJb14HoLKkU2AmN+Ngg3ZrOG 7a3Sg5Nd/JpVFhKy3hLf/OsHzkCYnDOCxzeNqBAjuNLPZSqjyvF2o5I66YktTDu6SQjL cHcJ82OElaj687msYmKybzjdxIqYF51ezL6Rd2BJ/HXPKKBEi6lSn9d1UW+aK86P8Dvk gIpA== MIME-Version: 1.0 From: Craig Ringer Date: Mon, 28 Oct 2019 05:53:00 -0000 Message-ID: Subject: Path resolution inconsistency between process("binary") and @var("var@cu","module") To: systemtap@sourceware.org Content-Type: text/plain; charset="UTF-8" X-IsSubscribed: yes X-SW-Source: 2019-q4/txt/msg00012.txt.bz2 Hi all TL;DR: @var("...","module") doesn't resolve "module" against PATH like process("execname") does; confusion ensures. SCCE included, fix proposed, comments requested. ---- I've recently discovered how amazing and powerful SystemTap is, after having it on my backburner to learn for years while struggling to achieve anything with linux-perf. I've been keeping notes on some suggestions to make for the documentation which I'll send separately. But first I've found (and worked around) an issue with user-space probing that I'd like to patch, but I'm looking for confirmation and advice first. The gist is that the PATH-resolution done by "process(...)" probes is not consistent with how @var(...) and @cast(...) resolve modules within stap functions called by probes. This causes stap to report very confusing failures to resolve symbols that are obviously right there as far as the user is concerned. The working cases are, for some probe on a function/statement in a userspace executable: - @var("sym@CU") directly in a probe; doesn't require module argument at all - @var("sym@CU", "/full/path/to/executable") in a function() called by a probe: finds var The failing cases are, all when in a function() called by a probe of a userspace executable: - @var("sym@CU") in the function: reports missing symbol, tries to resolve in "kernel" - @var("sym@CU","executable") in the function: also tries to resolve in kernel The probe type doesn't seem to matter that much: process("executable"), process("/path/to/executable"), process() and target mode, process(pid), etc etc all behave the same. @cast has similar issues to @var. SCCE with cases shown: program to probe: /* path_vs_atvar.c */ #include static int global_variable = 0; int main(int argc, char *argv[]) { global_variable = argc > 1 ? atoi(argv[1]) : 0; return global_variable; } /* end path_vs_atvar.c */ compilation: gcc -ggdb3 -Wall path_vs_atvar.c -o path_vs_atvar tapscript: function get_global() { // Doesn't work, preprocesses to @var(...,"kernel") [1] //return @var("global_variable@path_vs_atvar.c"); // Doesn't work, module argument treated as kernel [2] //return @var("global_variable@path_vs_atvar.c","path_vs_atvar"); // Fully qualified path as literal string: works, cumbersome/impractical [3] //return @var("global_variable@path_vs_atvar.c","/home/craig/projects/scrapcode/systemtap/path-vs-atvar/path_vs_atvar"); // Path as script-arg: works, though also cumbersome as user must resolve // and specify path as absolute path which defeats the point of path-expansion // in process probes, doesn't work well with -c, etc. [4] return @var("global_variable@path_vs_atvar.c",@1); } probe process.function("main").return { // Works: resolves variable in correct module for any suitable probe type like: // // process("path_vs_atvar") // process("/path/to/path_vs_atvar") // process($1) with path arg // process() using -c mode // // etc. [0] // //printf("global is %d\n", @var("global_variable@path_vs_atvar.c")); // Wheras when we indirect via a function, results vary printf("global is %d\n", get_global()); } To invoke, uncomment the [case] you want to try and run: sudo stap path_vs_atvar.stp -c ./path_vs_atvar for case [1] the result is: probe main@/home/craig/projects/scrapcode/systemtap/path-vs-atvar/path_vs_atvar.c:3 process=/home/craig/projects/scrapcode/systemtap/path-vs-atvar/path_vs_atvar reloc=.absolute pc=0x401126 semantic error: unresolved @var() expression: operator '@var' at path_vs_atvar.stp:4:12 thrown from: elaborate.cxx:6621 source: return @var("global_variable@path_vs_atvar.c"); ^ Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 262184virt/143804res/13596shr/129868data kb, in 60usr/120sys/187real ms. Missing separate debuginfos, use: debuginfo-install kernel-core-5.3.5-200.fc30.x86_64 You'd expect case [1] above, with no module-spec, to resolve the @var in the context the function is called in, but it doesn't. It's trying to use the "kernel" module per -vvv: return @var("global_variable@probe_vs_atvar.c", "kernel") semantic error: unresolved @var() expression: operator '@var' at path_vs_atvar.stp thrown from: elaborate.cxx:6621 source: return @var("global_variable@probe_vs_atvar.c"); so that's weird. The global is clearly there and in the specified CU: $ gdb -q -ex "info variables global_variable" -ex quit ./path_vs_atvar Reading symbols from ./path_vs_atvar... All variables matching regular expression "global_variable": File path_vs_atvar.c: 2: static int global_variable; $ But if you change it to explicitly reference the module "probe_vs_atvar" per [2] the outcome is the same, even though we *know* the module "path_vs_atvar" must exist and be accessible since we're being invoked from a .process probe on it. After all, it works when directly within the probe body [0]. If you use an absolute path to the module then the @var works within the function [4], e.g. @var("global_variable@path_vs_atvar.c","/path/to/path_vs_atvar"); but that's exceedingly impractical. The most tolerable workaround I found so far was to use a commandline argument to specify the full path to the executable. This makes process PATH resolution completely useless, but at least it works, like in case [5] in the SCCE. As I have quite a large set of executables to probe I tried to simplify use by using a macro to expand sub-paths from a basedir, but that doesn't work either. Implicit string concatenation doesn't get repeated after @MACRO expansion so @var("sym@CU", @MYBASEDIR "bin/foo") won't work. I didn't find any language support for preprocessing-time explicit string concatenation. So here I am. I did some code digging. It's my first time looking at the sytemtap code and I've only used it for a week, so please forgive errors/oversights, but AFAICS the important code is dwarf_atvar_expanding_visitor(...) and parse_atvar_op(...). I haven't quite worked out why the case in [0] works but I'm assuming it's something to do with focus_on_module(). For the function cases, I think this is partly due to the is_user_module() call in setupdwfl.cxx which assumes that userspace modules must always be absolute paths. bool is_user_module(const std::string &m) { return m[0] == '/' && m.rfind(".ko", m.length() - 1) != m.length() - 3; } That strikes me as an awfully bogus assumption; it should be possible to specify a userspace module by bare-name in @var, @cast, etc not just a kernel module. But taking a step back: systemtap appears to expand functions once per invocation in a probe anyway, so why doesn't the function benefit from the focus_on_module() of the probe and work like case [0] without any specification of module name at all? Phew! Ideas? -- Craig Ringer http://www.2ndQuadrant.com/ 2ndQuadrant - PostgreSQL Solutions for the Enterprise