* [Bug default/30274] New: RFE have fedabipkgdiff prefer debuginfod when available
@ 2023-03-26 1:40 woodard at redhat dot com
2023-03-26 3:18 ` [Bug default/30274] " fche at redhat dot com
2023-03-26 14:24 ` woodard at redhat dot com
0 siblings, 2 replies; 3+ messages in thread
From: woodard at redhat dot com @ 2023-03-26 1:40 UTC (permalink / raw)
To: libabigail
https://sourceware.org/bugzilla/show_bug.cgi?id=30274
Bug ID: 30274
Summary: RFE have fedabipkgdiff prefer debuginfod when
available
Product: libabigail
Version: unspecified
Status: NEW
Severity: normal
Priority: P2
Component: default
Assignee: dodji at redhat dot com
Reporter: woodard at redhat dot com
CC: libabigail at sourceware dot org
Target Milestone: ---
This came up as a continuation of PR#29911 one thing that FchE noticed was that
fedabipkgdiff was downloading debuginfo packages rather than fetching the data
from debuginfod even though libabigail being based on elfutils has the
capability to fetch debuginfod.
What we figured out was that somewhere in fedabipkgdiff it was deciding to and
fetching the debuginfod from koji and then it would prefer that over debuginfod
data. It was FchE that noticed this but once I started thinking about it, I
realized that this could greatly improve fedabipkgdiff. As I was running my
full distro test this is the kind of thing that I noticed in my top session as
I monitored progress. Note how many processes are running the cpio to unpack a
package vs how many jobs actually running abipkgdiff.
top - 18:30:34 up 1 day, 9:34, 4 users, load average: 31.53, 37.87, 42.06
Tasks: 581 total, 16 running, 565 sleeping, 0 stopped, 0 zombie
%Cpu(s): 39.6 us, 12.4 sy, 0.0 ni, 23.8 id, 24.0 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 128119.5 total, 65441.2 free, 5400.2 used, 58509.8 buff/cache
MiB Swap: 32124.0 total, 31700.5 free, 423.5 used. 122719.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3199639 ben 20 0 1635092 1.3g 131052 S 99.7 1.1 0:18.00
abipkgdiff
3199358 ben 20 0 722808 596612 47728 S 99.3 0.5 0:08.26
abipkgdiff
3183615 ben 20 0 189168 91344 12564 S 99.7 0.1 4:30.37
abipkgdiff
3087305 ben 20 0 268540 42156 15720 R 99.7 0.0 28:23.80 python
3155605 ben 20 0 266940 40540 15676 R 100.0 0.0 9:20.09 python
3199415 ben 20 0 266036 39764 15728 R 100.0 0.0 0:22.19 python
3195139 ben 20 0 265384 39100 15732 R 100.0 0.0 1:41.93 python
3197990 ben 20 0 265384 39084 15716 R 100.0 0.0 0:48.76 python
3156907 ben 20 0 265384 39076 15740 R 99.7 0.0 10:42.88 python
3199255 ben 20 0 265388 38964 15788 R 99.7 0.0 0:30.69 python
3172092 ben 20 0 265388 38896 15712 R 100.0 0.0 7:19.67 python
3185309 ben 20 0 265388 38868 15712 R 99.7 0.0 4:01.49 python
3188347 ben 20 0 264996 38720 15792 R 99.7 0.0 3:21.35 python
3196118 ben 20 0 265132 38656 15696 R 100.0 0.0 1:21.71 python
3081909 ben 20 0 264996 38616 15712 R 94.7 0.0 24:21.68 python
3199647 ben 20 0 264996 38584 15724 S 99.7 0.0 0:18.55 python
3200028 ben 20 0 20860 14364 4752 D 0.7 0.0 0:00.04 rpm2cpio
3200000 ben 20 0 20868 14328 4716 D 1.3 0.0 0:00.07 rpm2cpio
3199717 ben 20 0 20784 14260 4492 D 2.7 0.0 0:00.25 rpm2cpio
3191525 ben 20 0 20784 14216 4712 D 1.0 0.0 0:00.92 rpm2cpio
3199737 ben 20 0 20860 14212 4600 D 0.7 0.0 0:00.07 rpm2cpio
3189094 ben 20 0 20860 14168 4552 D 1.3 0.0 0:02.47 rpm2cpio
3199830 ben 20 0 20824 14148 4536 D 2.3 0.0 0:00.19 rpm2cpio
3200088 ben 20 0 20860 14104 4488 R 16.9 0.0 0:00.51 rpm2cpio
3199794 ben 20 0 20860 14064 4716 D 0.7 0.0 0:00.10 rpm2cpio
3192390 ben 20 0 20860 14008 4460 D 0.3 0.0 0:00.92 rpm2cpio
3199496 ben 20 0 20856 14004 4552 D 0.3 0.0 0:00.10 rpm2cpio
3192391 ben 20 0 221068 1172 1080 S 0.3 0.0 0:01.02 cpio
3191526 ben 20 0 221068 1168 1076 S 1.0 0.0 0:00.84 cpio
3199718 ben 20 0 221068 1160 1068 S 3.0 0.0 0:00.31 cpio
3189095 ben 20 0 221068 1156 1064 S 1.3 0.0 0:02.02 cpio
3200029 ben 20 0 221068 1156 1064 S 1.0 0.0 0:00.04 cpio
3199497 ben 20 0 221068 1136 1044 S 0.3 0.0 0:00.07 cpio
3199831 ben 20 0 221068 1132 1040 S 2.7 0.0 0:00.21 cpio
3200089 ben 20 0 221068 1132 1040 R 18.3 0.0 0:00.55 cpio
3199795 ben 20 0 221068 1108 1012 S 0.3 0.0 0:00.09 cpio
3200001 ben 20 0 221068 1108 1012 S 1.7 0.0 0:00.09 cpio
20964 root 20 0 0 0 0 S 0.3 0.0 0:47.66 NFSv4
callback
2612147 root 20 0 0 0 0 I 0.3 0.0 0:00.51
kworker/9:2-events_freezable
2858412 root 20 0 0 0 0 I 0.3 0.0 0:05.50
kworker/u256:6-events_unbound
2953630 root 20 0 0 0 0 I 0.3 0.0 0:04.37
kworker/u256:0-rpciod
3096956 root 20 0 0 0 0 I 0.3 0.0 0:01.41
kworker/u256:5-events_unbound
3182633 root 20 0 0 0 0 I 0.3 0.0 0:00.34
kworker/u256:9-rpciod
Furthermore fedabipkgdiff creates a cache directory of rpms. I have frequently
had test terminate because I've run out of space in this directory. A notable
percentage of the files that end up in the cache directory are the debuginfo
packages.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 3+ messages in thread
* [Bug default/30274] RFE have fedabipkgdiff prefer debuginfod when available
2023-03-26 1:40 [Bug default/30274] New: RFE have fedabipkgdiff prefer debuginfod when available woodard at redhat dot com
@ 2023-03-26 3:18 ` fche at redhat dot com
2023-03-26 14:24 ` woodard at redhat dot com
1 sibling, 0 replies; 3+ messages in thread
From: fche at redhat dot com @ 2023-03-26 3:18 UTC (permalink / raw)
To: libabigail
https://sourceware.org/bugzilla/show_bug.cgi?id=30274
--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> ---
fedabipkgdiff does a busy loop on waitpid() on the abipkgdiff child process:
@log_call
def abipkgdiff(cmp_half1, cmp_half2):
[...]
proc = subprocess.Popen(' '.join(cmd), shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
# So we could have done: stdout, stderr = proc.communicate()
# But then the documentatin of proc.communicate says:
#
# Note: The data read is buffered in memory, so do not use this
# method if the data size is large or unlimited. "
#
# In practice, we are seeing random cases where this
# proc.communicate() function does *NOT* terminate and seems to be
# in a deadlock state. So we are avoiding it altogether. We are
# then busy looping, waiting for the spawn process to finish, and
# then we get its output.
#
while True:
if proc.poll() != None:
break
Not sure that interpretation of python docs is correct, but even then, a wee
bit of sleep wouldn't hurt.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 3+ messages in thread
* [Bug default/30274] RFE have fedabipkgdiff prefer debuginfod when available
2023-03-26 1:40 [Bug default/30274] New: RFE have fedabipkgdiff prefer debuginfod when available woodard at redhat dot com
2023-03-26 3:18 ` [Bug default/30274] " fche at redhat dot com
@ 2023-03-26 14:24 ` woodard at redhat dot com
1 sibling, 0 replies; 3+ messages in thread
From: woodard at redhat dot com @ 2023-03-26 14:24 UTC (permalink / raw)
To: libabigail
https://sourceware.org/bugzilla/show_bug.cgi?id=30274
--- Comment #2 from Ben Woodard <woodard at redhat dot com> ---
One thing that I can say from experience is when there are a lot of
differences, the size of output files is a few hundred thousand lines and
probably amounts to a couple of megabytes of text. This is tiny in relation to
the size of memory used by abipkgdiff itself when building the corpus. (I've
seen up to 25G just casually watching top.) So to me it sounds like the concern
that led to that design decision is unwarranted.
--
You are receiving this mail because:
You are on the CC list for the bug.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-03-26 14:24 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-26 1:40 [Bug default/30274] New: RFE have fedabipkgdiff prefer debuginfod when available woodard at redhat dot com
2023-03-26 3:18 ` [Bug default/30274] " fche at redhat dot com
2023-03-26 14:24 ` woodard at redhat dot com
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).