[Bug default/29303] New: Unacceptable performance doing fedabipkgdiff on some packages

[Bug default/29303] New: Unacceptable performance doing fedabipkgdiff on some packages

[woodard at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

            Bug ID: 29303
           Summary: Unacceptable performance doing fedabipkgdiff on some
                    packages
           Product: libabigail
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: default
          Assignee: dodji at redhat dot com
          Reporter: woodard at redhat dot com
                CC: libabigail at sourceware dot org
  Target Milestone: ---

Created attachment 14184
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14184&action=edit
backtraces from several of the processes

The run time for doing self-compare's for a few packages is unacceptable. These
were running for more than 11hrs. With the 2.0 version of libabigail their
runtimes are more reasonable.

1125479 pts/3    R    819:36 python
/home/ben/Shared/Work/test/libabigail-x86_64/bin//fedabipkgdiff --self-compare
-a --from fc36 dovecot
1196728 pts/3    R    787:03 python
/home/ben/Shared/Work/test/libabigail-x86_64/bin//fedabipkgdiff --self-compare
-a --from fc36 libdb
1259587 pts/3    R    884:13 python
/home/ben/Shared/Work/test/libabigail-x86_64/bin//fedabipkgdiff --self-compare
-a --from fc36 llvm11
1267329 pts/3    R    815:42 python
/home/ben/Shared/Work/test/libabigail-x86_64/bin//fedabipkgdiff --self-compare
-a --from fc36 lldb
1310898 pts/3    R    763:44 python
/home/ben/Shared/Work/test/libabigail-x86_64/bin//fedabipkgdiff --self-compare
-a --from fc36 mozjs91
1404655 pts/3    R    675:46 python
/home/ben/Shared/Work/test/libabigail-x86_64/bin//fedabipkgdiff --self-compare
-a --from fc36 webkit2gtk3

Attached are backtraces from the running processes.

These are hard to read and don't look immediately similar but one thing did
jump out at me looking at them was:

$ grep _M_erase backtraces.txt 
#1  std::_Hashtable<abigail::ir::class_or_union const*,
abigail::ir::class_or_union const*, std::allocator<abigail::ir::class_or_union
const*>, std::__detail::_Identity, std::equal_to<abigail::ir::class_or_union
const*>, std::hash<abigail::ir::class_or_union const*>,
std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash,
std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false,
true, true> >::_M_erase (__n=0x7fb1df831c60, __prev_n=0x7fb354000cf8,
__bkt=<optimized out>, this=0x7fb354000ce8) at
/usr/include/c++/12/bits/hashtable.h:2316
#2  std::_Hashtable<abigail::ir::class_or_union const*,
abigail::ir::class_or_union const*, std::allocator<abigail::ir::class_or_union
const*>, std::__detail::_Identity, std::equal_to<abigail::ir::class_or_union
const*>, std::hash<abigail::ir::class_or_union const*>,
std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash,
std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false,
true, true> >::_M_erase(std::integral_constant<bool, true>,
abigail::ir::class_or_union const* const&) [clone .constprop.0] [clone .isra.0]
(this=0x7fb354000ce8, __k=@0x7fb3598b0460: 0x7fb31b0da190) at
/usr/include/c++/12/bits/hashtable.h:2370
#1  std::_Hashtable<abigail::ir::class_or_union const*,
abigail::ir::class_or_union const*, std::allocator<abigail::ir::class_or_union
const*>, std::__detail::_Identity, std::equal_to<abigail::ir::class_or_union
const*>, std::hash<abigail::ir::class_or_union const*>,
std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash,
std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false,
true, true> >::_M_erase (__n=0x7fb1df831c60, __prev_n=0x7fb354000cf8,
__bkt=<optimized out>, this=0x7fb354000ce8) at
/usr/include/c++/12/bits/hashtable.h:2316
#2  std::_Hashtable<abigail::ir::class_or_union const*,
abigail::ir::class_or_union const*, std::allocator<abigail::ir::class_or_union
const*>, std::__detail::_Identity, std::equal_to<abigail::ir::class_or_union
const*>, std::hash<abigail::ir::class_or_union const*>,
std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash,
std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false,
true, true> >::_M_erase(std::integral_constant<bool, true>,
abigail::ir::class_or_union const* const&) [clone .constprop.0] [clone .isra.0]
(this=0x7fb354000ce8, __k=@0x7fb3598b0460: 0x7fb31b0da190) at
/usr/include/c++/12/bits/hashtable.h:2370

4 out of 5 of them were in the process of some sort of hash table erase
function. That seems suspicious to me.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[woodard at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

Ben Woodard <woodard at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |27019


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=27019
[Bug 27019] BUILD metabug
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

dodji at redhat dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

--- Comment #1 from dodji at redhat dot com ---
This should now be fixed by the patch
https://sourceware.org/git/?p=libabigail.git;a=commit;h=69795d1bf8af9663fb3188d3e70cd72bc0857457.

Thanks and sorry for the inconvenience.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[woodard at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

--- Comment #2 from Ben Woodard <woodard at redhat dot com> ---
This greatly improved the performance for some packages but others continue to
take an unacceptable amount of time:

2084669 pts/3    R    227:05 python fedabipkgdiff --self-compare -a --from fc36
cmake
2162224 pts/3    R    226:40 python fedabipkgdiff --self-compare -a --from fc36
gcc
2267972 pts/3    R    292:05 python fedabipkgdiff --self-compare -a --from fc36
lldb
2269176 pts/3    R    291:14 python fedabipkgdiff --self-compare -a --from fc36
llvm11
2292442 pts/3    R    150:57 python fedabipkgdiff --self-compare -a --from fc36
mozjs68
2292459 pts/3    R    210:07 python fedabipkgdiff --self-compare -a --from fc36
mozjs78
2292732 pts/3    R    201:26 python fedabipkgdiff --self-compare -a --from fc36
mozjs91
2348051 pts/3    R     59:18 python fedabipkgdiff --self-compare -a --from fc36
qt6-qtdeclarative
2388701 pts/3    S     37:03 python fedabipkgdiff --self-compare -a --from fc36
vtk
2389651 pts/3    R     61:44 python fedabipkgdiff --self-compare -a --from fc36
webkit2gtk3

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[woodard at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

--- Comment #3 from Ben Woodard <woodard at redhat dot com> ---
Other packages that apparently got stuck over night. These include ones that
are not just C++

  34212 pts/1    R    966:44 fedabipkgdiff --self-compare -a --from fc36
zlib-ada
  54036 pts/1    R    1331:22 fedabipkgdiff --self-compare -a --from fc36
xorg-x11-server-Xorg
 112666 pts/1    R    1182:30 fedabipkgdiff --self-compare -a --from fc36
webkit2gtk3
 133157 pts/1    R    911:28 fedabipkgdiff --self-compare -a --from fc36
vim-minimal
 152053 pts/1    R    1193:04 fedabipkgdiff --self-compare -a --from fc36
util-linux-core
 240094 pts/1    R    818:36 fedabipkgdiff --self-compare -a --from fc36
systemd-udev
 397285 pts/1    R    531:24 fedabipkgdiff --self-compare -a --from fc36
samba-winbind
 397346 pts/1    R    732:20 fedabipkgdiff --self-compare -a --from fc36
samba-libs
 398027 pts/1    R    730:44 fedabipkgdiff --self-compare -a --from fc36
samba-client
 508636 pts/1    R    252:53 fedabipkgdiff --self-compare -a --from fc36 qt-x11

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

dodji at redhat dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends on|                            |29360

--- Comment #4 from dodji at redhat dot com ---
So, the fedabipkgdiff --self-compare -a --from fc36 zlib-ada one appears to be
due to a lack of support for the "range type", in Ada.  That is what leads to
an infinite loop, sadly.

I have filed a separate tracking bug for it at #29360.  Also, I have started
the work on that in the branch
https://sourceware.org/git/?p=libabigail.git;a=shortlog;h=refs/heads/free-range-types.

The current status of that branch transforms the infinite loop into an assert
now, at least.  Maybe we could just merge it as is, for now, I am not sure.

As this is a lower priority, I'll move to looking into more pressing stuff.

This is however a nice little project of its own to work on, maybe for someone
who would like to get into libabigail hacking? ;-)


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=29360
[Bug 29360] Ada ranges emitting/reading/diffing are not supported in libabigail
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

--- Comment #5 from dodji at redhat dot com ---
(In reply to dodji from comment #4)
> So, the fedabipkgdiff --self-compare -a --from fc36 zlib-ada one appears to
> be due to a lack of support for the "range type", in Ada.  That is what
> leads to an infinite loop, sadly.
> 
> I have filed a separate tracking bug for it at #29360.

I meant this one: https://sourceware.org/bugzilla/show_bug.cgi?id=29360

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[woodard at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

--- Comment #6 from Ben Woodard <woodard at redhat dot com> ---
In my latest test run with trunk as of df28c220976f41620b0bd7b22000815e11f66b75

Testing the entirety of Fedora 36 took 2 days and 4 hours on 10 cores.

2d+4h=52hrs * 10 cores = 520 compute hrs
of those 169 timed out at 60 min so:
24029 jobs - 169 = 23860 jobs
520 hrs - 169 = 351 hrs
Therefore 23869 jobs took 351 hrs or 0.88s each on average while those 169 took
about 33% of the time.

These are the packages which took more than 1hr of runtime:
pcp-libs                        mozjs68                         zlib-ada
sssd-idp                        aunit                           qemu-img
gprbuild                        percona-xtrabackup              qemu-common
lldb                            pcb                             cbmc
glusterfs-fuse                  emacs-common                    python3-openmpi
lvm2                            kea                             clang-devel
gmsh                            libvirt-daemon                  gimp-libs
vtk                             opencascade-modeling            qwt5-qt5
mingw-binutils-generic          mesa-libxatracker               wesnoth
git-core                        boost-regex                     ceres-solver
iaito                           clang-libs                      Singular
qtwebkit                        lapack                          ghc9.4
kismet                          ghc-data-memocombinators        systemd-libs
libreoffice-ure                 gnome-terminal-nautilus         gcc-gfortran
opencascade-ocaf                qt5-qtwebkit                    PragmARC
mesa-libglapi                   ghc-tar-conduit                 ompl
libreoffice-pdfimport           flang                           freecad
libreoffice-core                net-snmp-libs                   prusa-slicer
rstudio                         qpdf-libs                       smesh
ceph                            ghc-call-stack                  cmake
ugene                           openblas-threads64_             psblas3
tuxpaint                        ydotool                         pcp
opencryptoki                    ghc-mmap                        firebird
libreoffice-pyuno               e2fsprogs-libs                  mesa-libOSMesa
gnome-builder                   opencascade-foundation          vfrnav
hatari                          denemo                          llvm-libs
octave                          gstreamer1-plugins-good-gtk     kdebase3
qemu-virtiofsd                  ghc-data-inttrie                kvirc
qt5-qt3d                        ghc-x509-system                 matreshka
sundials                        ghc-typed-process              
systemd-container
samba-client                    Agda                            llvm11
webkit2gtk3-jsc                 ghc-ConfigFile                  widelands
gedit                           kdeplasma-addons                openmsx
llvm7.0                         gnumeric                        qt3
opencascade-visualization       ghc-lens-family-core            boost
alliance                        gvfs-fuse                       mesa-libGL
florist                         ghc-filtrable                   webkit2gtk3
cross-gcc                       ghc-network-byte-order          bind-utils
binutils-gold                   realtime-setup                 
libreoffice-xsltfilter
libblockdev-mdraid              ghc-unliftio-core               mlir
aws                             ghc-html-conduit                frozen-bubble
systemd-udev                    ghc-gi-ostree                   ahven
blender                         gimp-dbp                        llvm11-libs
kf5-khtml                       ghc-prettyprinter-ansi-terminal samba-winbind
libreoffice-gtk3                java-17-openjdk-headless       
arm-none-eabi-gcc-cs
samba-libs                      anet                            mine_detector
flexiblas-netlib64              llvm12                          hpx
mesa-libEGL                     lvm2-libs                       vxl
MUMPS                           boost-wave                      llvm13-libs
xorg-x11-server-Xorg            mozjs78                         pingus
gcc-c++                         cataclysm-dda                   qt5-qtbase-gui
lammps                          R-Biostrings                    mozjs91
qt-x11                          paraview                        llvm8.0
mesa                            mscore                          psi4
libblockdev-fs                  java-11-openjdk-headless       
qt6-qtdeclarative
gcc                             systemd-networkd
mesa-libgbm                     qgis

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/29303] Unacceptable performance doing fedabipkgdiff on some packages

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=29303

dodji at redhat dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends on|29360                       |


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=29360
[Bug 29360] Ada ranges emitting/reading/diffing are not supported in libabigail
-- 
You are receiving this mail because:
You are on the CC list for the bug.