From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 103953 invoked by alias); 24 Jan 2018 08:49:06 -0000 Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org Received: (qmail 103893 invoked by uid 89); 24 Jan 2018 08:48:59 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.4 required=5.0 tests=BAYES_00,KAM_NUMSUBJECT,RCVD_IN_DNSWL_NONE,SPF_PASS autolearn=no version=3.3.2 spammy=HTo:U*systemtap, reactor, HX-Received:Wed, HContent-Transfer-Encoding:8bit X-HELO: mail-wm0-f48.google.com Received: from mail-wm0-f48.google.com (HELO mail-wm0-f48.google.com) (74.125.82.48) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 24 Jan 2018 08:48:58 +0000 Received: by mail-wm0-f48.google.com with SMTP id 143so6816070wma.5 for ; Wed, 24 Jan 2018 00:48:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:organization:message-id:date :user-agent:mime-version:content-transfer-encoding:content-language; bh=PL+0NKoOKpaEF6fvgJpx6hHixj0MFLyQaufEybx7crc=; b=VAbemafRrxFf0xwnDD7n408hHjRuvkoW60C2xsT6dKvm+4xYej+qhPSC3VtsyBQvk1 dK271ruqRFx/o9/jj2PcYC5wzVWeZdcJT5gXF4Ic/3SgSyoDXrRkw1xtwXyDp55qcrmA byC8+UNtm6DEz514GQMzpRLFHGmbboURerjWJvi9GeDEn5T0zSPhG+cgh5NuX2F2Bvzp 3K8UYlRhNLyqmZSUIgsta0aqkRRClj2jqS6aVhPHvMwKvZuS12DJjXE+UL/kjvNFlWu8 96lZFZl4pTQXJ2jffn2EkD0+jsQ4zzSEJcS8ZWZV0985GUH/1clIDm1HZaKY83GB7d0N e8ag== X-Gm-Message-State: AKwxytdnxA1Jf99Af+ZRpVv8PwLAUru6g8Gk1MzU5zvmAHF1ZuCVEb9a T/+p/Bc6XnU+Bd+FZpykxRwsSA== X-Google-Smtp-Source: AH8x226UUl0l2gS+QvnkOI8wu3arUVnu7GnxA/ORgYpn/MrtpDN3BMxtiDAy8N4JHVjvxG29C06EfQ== X-Received: by 10.80.204.1 with SMTP id m1mr23001547edi.146.1516783735844; Wed, 24 Jan 2018 00:48:55 -0800 (PST) Received: from avi.cloudius-systems.com ([199.203.229.89]) by smtp.gmail.com with ESMTPSA id l23sm9379784edc.20.2018.01.24.00.48.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Jan 2018 00:48:54 -0800 (PST) To: systemtap@sourceware.org From: Avi Kivity Subject: Systemtap kernel backtraces not working on 4.14.14 Message-ID: <4e07a614-b734-8d91-b18d-d6313981c191@scylladb.com> Date: Wed, 24 Jan 2018 08:49:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-IsSubscribed: yes X-SW-Source: 2018-q1/txt/msg00008.txt.bz2 I have a centos installation with 4.14.14 installed. Since the systemtap that came in the box wouldn't work with that kernel, I built git HEAD. A script that calls print_syms() used to display a backtrace, but now only shows systemtap kmod symbols and one entry from the kernel proper:   at:       8483   to: rcu_sched  0xffffffffa0335150 : 0xffffffffa0335150 [stap_fc3e91a0b756c0a804d73b3d051959b6__1891+0xd150/0x0]  0xffffffffa0336476 : 0xffffffffa0336476 [stap_fc3e91a0b756c0a804d73b3d051959b6__1891+0xe476/0x0]  0xffffffffa0336eb4 : 0xffffffffa0336eb4 [stap_fc3e91a0b756c0a804d73b3d051959b6__1891+0xeeb4/0x0]  0xffffffffa0339ecc : 0xffffffffa0339ecc [stap_fc3e91a0b756c0a804d73b3d051959b6__1891+0x11ecc/0x0]  0xffffffffa0328014 : 0xffffffffa0328014 [stap_fc3e91a0b756c0a804d73b3d051959b6__1891+0x14/0x0]  0xffffffff817db0a5 : __schedule+0x395/0x880 [kernel]  0x0 : 0x0 Here is the script: global mon, trace, header, nr_trace, start_system_time # mon[tid()] = 1    if it's a reactor thread that we're tracing # mon[tid()] = 0        if we don't care about this thread # trace[tid(), idx]     multiline trace of recent events in rid() # header[tid(), idx]    header for trace above # nr_trace[tid()]       number of lines in trace probe syscall.io_submit {     if (cpu() != 7) { next }     t = tid()     mon[t] = 1     start_system_time[t] = task_stime()     ts = sprintf("%d", t)     start_stopwatch(ts) } function add_trace(task_id, other_label, other_name) {     ts = sprintf("%d", task_id)     idx = nr_trace[task_id]++     header[task_id, idx] = sprintf("at: %10d %s: %s", read_stopwatch_us(ts), other_label, other_name)     trace[task_id, idx] = backtrace() } probe scheduler.ctxswitch {     if (cpu() != 7) { next }     from = prev_tid     to = next_tid     if (mon[from]) {         add_trace(from, "from", next_task_name)     }     if (mon[to]) {         add_trace(to, "  to", prev_task_name)     } } probe syscall.io_submit.return {     if (cpu() != 7) { next }     t = tid()     mon[t] = 0     ts = sprintf("%d", t)     tm = read_stopwatch_ms(ts)     systime = cputime_to_msecs(task_stime() - start_system_time[t])     if (t in nr_trace) {         nr = nr_trace[t]         if (tm >= 10) {             printf("wall %6d ms sys %6d ms\n", tm, systime)             for (idx = 0; idx != nr; ++idx) {                printf("  %s\n", header[t, idx])                print_syms(trace[t, idx])             }         }         for (idx = 0; idx != nr; ++idx) {            delete trace[t, idx]            delete header[t, idx]         }         delete nr_trace[t]     }     delete_stopwatch(ts) } Is there something in the kernel configuration I need to turn on (.config taken from elrepo's -ml kernel)? Or a missing dependency while building systemtap? Maybe systemtap can't cope with retpolines?