From: "dsmith at redhat dot com" <sourceware-bugzilla@sourceware.org>
To: systemtap@sourceware.org
Subject: [Bug runtime/20735] "soft lockup" bug on RHEL7 ppc64
Date: Tue, 25 Oct 2016 21:17:00 -0000 [thread overview]
Message-ID: <bug-20735-6586-uKw9NLRCcZ@http.sourceware.org/bugzilla/> (raw)
In-Reply-To: <bug-20735-6586@http.sourceware.org/bugzilla/>
https://sourceware.org/bugzilla/show_bug.cgi?id=20735
--- Comment #1 from David Smith <dsmith at redhat dot com> ---
I've been looking at this lockup, and my suspicion is that the problem lies
with the systemtap runtime grabbing the module_mutex. The module_mutex is
exported by the kernel, and the systemtap runtime grabs that mutex when calling
kallsyms_on_each_symbol(). In the lockup below, there are at least 5 systemtap
modules loaded simultaneously. When the last systemtap module got loaded, that
triggered the other 4 to get notified that a new module was loaded. That causes
the other 4 to try to grab the module_mutex to update module symbols (via
stapkp_refresh() in runtime/linux/kprobes.c). In addition, the last one loaded
could be calling stapkp_init() (runtime/linux/kprobes.c) which also tries to
grab the module mutex.
So, my suspicion is that all 5 of those modules are trying to grab that same
mutex at the same time. Plus, kallsyms_on_each_symbol() isn't quick, since it
calls a function on every kernel symbol.
Here's the comment before the module_mutex definition in kernel/module.c:
/*
* Mutex protects:
* 1) List of modules (also safely readable with preempt_disable),
* 2) module_use links,
* 3) module_addr_min/module_addr_max.
* (delete uses stop_machine/add uses RCU list operations). */
Based on that comment, I'm going to try to see what happens if we disable
preemption before calling kallsyms_on_each_symbol() and enabling preemption
afterwards instead of using module_mutex.
--
You are receiving this mail because:
You are the assignee for the bug.
next prev parent reply other threads:[~2016-10-25 21:17 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-24 19:51 [Bug runtime/20735] New: " dsmith at redhat dot com
2016-10-25 21:17 ` dsmith at redhat dot com [this message]
2016-11-03 14:56 ` [Bug runtime/20735] " dsmith at redhat dot com
2016-11-10 16:02 ` dsmith at redhat dot com
2023-05-06 21:40 ` agentzh at gmail dot com
2023-05-08 12:43 ` dsmith at redhat dot com
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bug-20735-6586-uKw9NLRCcZ@http.sourceware.org/bugzilla/ \
--to=sourceware-bugzilla@sourceware.org \
--cc=systemtap@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).