public inbox for glibc-bugs@sourceware.org help / color / mirror / Atom feed
* [Bug libc/26179] New: _dl_map_object_deps re-walking transitive deps of already loaded DSOs @ 2020-06-28 0:27 andrewjcg at gmail dot com 2020-06-28 0:28 ` [Bug libc/26179] " andrewjcg at gmail dot com ` (2 more replies) 0 siblings, 3 replies; 4+ messages in thread From: andrewjcg at gmail dot com @ 2020-06-28 0:27 UTC (permalink / raw) To: glibc-bugs https://sourceware.org/bugzilla/show_bug.cgi?id=26179 Bug ID: 26179 Summary: _dl_map_object_deps re-walking transitive deps of already loaded DSOs Product: glibc Version: 2.33 Status: UNCONFIRMED Severity: normal Priority: P2 Component: libc Assignee: unassigned at sourceware dot org Reporter: andrewjcg at gmail dot com CC: drepper.fsp at gmail dot com Target Milestone: --- Created attachment 12665 --> https://sourceware.org/bugzilla/attachment.cgi?id=12665&action=edit ccccccccccccfrheljkkugnelnetrrjlvlctrflrejtv In workflows with lots of DSOs which are independently `dlopen`d but which have several common dependency DSOs, each `_dl_map_object_deps` call will re-walk these common transitive deps, doing O(N) `strcmp`s in `_dl_map_object` for each one, which can become costly. I noticed this while looking into some workflows where Python code `dlopen`s a lot of DSOs, each of which shares a lot of transitive deps with the others. Profiling shows a significant time is spent in `strcmp`. It looks like much of this work can be avoided having each call to `_dl_map_object` also return whether that DSO was already loaded, and terminate the recursion if so. An example of an attempt at this is attached which removed most of this overhead (but I'm not entirely sure it's safe). -- You are receiving this mail because: You are on the CC list for the bug. ^ permalink raw reply [flat|nested] 4+ messages in thread
* [Bug libc/26179] _dl_map_object_deps re-walking transitive deps of already loaded DSOs 2020-06-28 0:27 [Bug libc/26179] New: _dl_map_object_deps re-walking transitive deps of already loaded DSOs andrewjcg at gmail dot com @ 2020-06-28 0:28 ` andrewjcg at gmail dot com 2020-07-03 16:06 ` P at draigBrady dot com 2020-07-03 16:18 ` carlos at redhat dot com 2 siblings, 0 replies; 4+ messages in thread From: andrewjcg at gmail dot com @ 2020-06-28 0:28 UTC (permalink / raw) To: glibc-bugs https://sourceware.org/bugzilla/show_bug.cgi?id=26179 Andrew <andrewjcg at gmail dot com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #12665|ccccccccccccfrheljkkugnelne |fix_dl_map_object_deps description|trrjlvlctrflrejtv | -- You are receiving this mail because: You are on the CC list for the bug. ^ permalink raw reply [flat|nested] 4+ messages in thread
* [Bug libc/26179] _dl_map_object_deps re-walking transitive deps of already loaded DSOs 2020-06-28 0:27 [Bug libc/26179] New: _dl_map_object_deps re-walking transitive deps of already loaded DSOs andrewjcg at gmail dot com 2020-06-28 0:28 ` [Bug libc/26179] " andrewjcg at gmail dot com @ 2020-07-03 16:06 ` P at draigBrady dot com 2020-07-03 16:18 ` carlos at redhat dot com 2 siblings, 0 replies; 4+ messages in thread From: P at draigBrady dot com @ 2020-07-03 16:06 UTC (permalink / raw) To: glibc-bugs https://sourceware.org/bugzilla/show_bug.cgi?id=26179 Pádraig Brady <P at draigBrady dot com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |P at draigBrady dot com -- You are receiving this mail because: You are on the CC list for the bug. ^ permalink raw reply [flat|nested] 4+ messages in thread
* [Bug libc/26179] _dl_map_object_deps re-walking transitive deps of already loaded DSOs 2020-06-28 0:27 [Bug libc/26179] New: _dl_map_object_deps re-walking transitive deps of already loaded DSOs andrewjcg at gmail dot com 2020-06-28 0:28 ` [Bug libc/26179] " andrewjcg at gmail dot com 2020-07-03 16:06 ` P at draigBrady dot com @ 2020-07-03 16:18 ` carlos at redhat dot com 2 siblings, 0 replies; 4+ messages in thread From: carlos at redhat dot com @ 2020-07-03 16:18 UTC (permalink / raw) To: glibc-bugs https://sourceware.org/bugzilla/show_bug.cgi?id=26179 Carlos O'Donell <carlos at redhat dot com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |carlos at redhat dot com Resolution|--- |DUPLICATE Status|UNCONFIRMED |RESOLVED --- Comment #1 from Carlos O'Donell <carlos at redhat dot com> --- (In reply to Andrew from comment #0) > Created attachment 12665 [details] > fix_dl_map_object_deps > > In workflows with lots of DSOs which are independently `dlopen`d but which > have several common dependency DSOs, each `_dl_map_object_deps` call will > re-walk these common transitive deps, doing O(N) `strcmp`s in > `_dl_map_object` for each one, which can become costly. > > I noticed this while looking into some workflows where Python code `dlopen`s > a lot of DSOs, each of which shares a lot of transitive deps with the > others. Profiling shows a significant time is spent in `strcmp`. > > It looks like much of this work can be avoided having each call to > `_dl_map_object` also return whether that DSO was already loaded, and > terminate the recursion if so. An example of an attempt at this is attached > which removed most of this overhead (but I'm not entirely sure it's safe). We are already working on resolving this as part of bug 17645. We are only v2.1 of the fix that switches to a full DFS RPO sort with "use" tracking per link map to avoid visiting nodes again, and terminating the sort early if the list was already sorted. I'm marking this as a duplicate of bug 17645. *** This bug has been marked as a duplicate of bug 17645 *** -- You are receiving this mail because: You are on the CC list for the bug. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-07-03 16:18 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-06-28 0:27 [Bug libc/26179] New: _dl_map_object_deps re-walking transitive deps of already loaded DSOs andrewjcg at gmail dot com 2020-06-28 0:28 ` [Bug libc/26179] " andrewjcg at gmail dot com 2020-07-03 16:06 ` P at draigBrady dot com 2020-07-03 16:18 ` carlos at redhat dot com
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).