[Bug default/19081] New: Make abipkgdiff be multi-threaded

[Bug default/19081] New: Make abipkgdiff be multi-threaded

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

            Bug ID: 19081
           Summary: Make abipkgdiff be multi-threaded
           Product: libabigail
           Version: unspecified
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: default
          Assignee: dodji at redhat dot com
          Reporter: dodji at redhat dot com
                CC: libabigail at sourceware dot org
  Target Milestone: ---

abipkgdiff can use pthreads to run comparison on pairs of libraries (each pair
contains two versions of the same library) carried by two packages, just like
what test-read-dwarf.cc does already.

That could potentially greatly speed up abipkgdiff, depending on the number of
physical cores the system has.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

--- Comment #6 from dodji at redhat dot com ---
Thanks!

I have started to review the patch on the mailing of the project.

You can read the review at
https://sourceware.org/ml/libabigail/2015-q4/msg00143.html.

Thanks again for spending time on this.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

--- Comment #2 from dodji at redhat dot com ---
Note that the .abignore file to take in account is the one present in the
latest package of the two being compared.

If the first package has an .abignore file and not the second one, then the
.abignore file of the first package is ignored.

Note also that additional suppression specification files can be passed to the
command line of the tool.  They are going to be taken in account in *addition*
to the *.abignore file found in the second package.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[ooprala at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

Ondrej Oprala <ooprala at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #8 from Ondrej Oprala <ooprala at redhat dot com> ---
Resolved by upstream commit 17b04f2e

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

--- Comment #1 from dodji at redhat dot com ---
Note that for each pair of binaries to compare in a pair of packages, the
reporting phase should not be done in parallel with the reporting phases of the
other pairs of binaries.

All the reporting of all pair of binaries should happen sequentially after the
reading, comparison and diff categorization (the phase where the system learns
about what diff report to filter out) happened in parallel mode.

So we might need to add entry points in the library to perform the diff
categorization separately from the reporting itself.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[ooprala at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

--- Comment #5 from Ondrej Oprala <ooprala at redhat dot com> ---
So here's my take on this issue.
I've been using the rpms of vtk to test this throughout.
Prior to any threading attempts, running this:
abipkgdiff vtk-6.1.0-26.fc23.x86_64.rpm vtk-6.2.0-8.fc23.x86_64.rpm --d1 
vtk-debuginfo-6.1.0-26.fc23.x86_64.rpm --d2 
vtk-debuginfo-6.2.0-8.fc23.x86_64.rpm

took a bit less than 4 mins:
230,08s user 6,60s system 101% cpu 3:53,03 total

threads make the numbers much nicer \o/
381,17s user 14,15s system 492% cpu 1:20,23 total

I tried to be as verbose as possible with the commit message, but if you 
think I missed
anything anywhere, please tell me, I want this to be as good as it can get.

Helgrind only complains about elf_version (and elf_begin), which we 
already discussed on the IRC.
Didn't try testing it with anything really huge yet like Ben's libgcj etc...
Any and all comments would be very appreciated.

On 07.10.2015 11:34, dodji at redhat dot com wrote:
> https://sourceware.org/bugzilla/show_bug.cgi?id=19081
>
>              Bug ID: 19081
>             Summary: Make abipkgdiff be multi-threaded
>             Product: libabigail
>             Version: unspecified
>              Status: NEW
>            Severity: enhancement
>            Priority: P2
>           Component: default
>            Assignee: dodji at redhat dot com
>            Reporter: dodji at redhat dot com
>                  CC: libabigail at sourceware dot org
>    Target Milestone: ---
>
> abipkgdiff can use pthreads to run comparison on pairs of libraries (each pair
> contains two versions of the same library) carried by two packages, just like
> what test-read-dwarf.cc does already.
>
> That could potentially greatly speed up abipkgdiff, depending on the number of
> physical cores the system has.
>

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

--- Comment #3 from dodji at redhat dot com ---
(In reply to dodji from comment #2)
> Note that the .abignore file to take in account is the one present in the
> latest package of the two being compared.
> 
> If the first package has an .abignore file and not the second one, then the
> .abignore file of the first package is ignored.
> 
> Note also that additional suppression specification files can be passed to
> the command line of the tool.  They are going to be taken in account in
> *addition* to the *.abignore file found in the second package.

Forget this comment, please.  It was meant to be for bug
https://sourceware.org/bugzilla/show_bug.cgi?id=19082, not this one, sorry.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

--- Comment #7 from dodji at redhat dot com ---
(In reply to dodji from comment #6)

> I have started to review the patch on the mailing of the project.
> You can read the review at
> https://sourceware.org/ml/libabigail/2015-q4/msg00143.html.

So let's continue the discussion on the mailing list.

Cheers.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[woodard at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

Ben Woodard <woodard at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |woodard at redhat dot com

--- Comment #4 from Ben Woodard <woodard at redhat dot com> ---
Something that sounds related and might be the same thing which would be
helpful in abidiff would be to make the reading of the libraries run in
parallel. This could be a great optimization especially if you allowed multiple
comparisons.

abidiff dir1/libfoo.so dir2/libfoo.so dir3/libfoo.so dir4/libfoo.so

The reading of the ABI could be done in parallel and then whether the
comparison is done in parallel or sequentially the output would have to be
serialized. However, doing it this way would need to have two variations one
which would generate in effect:

abidiff dir1/libfoo.so dir2/libfoo.so
abidiff dir1/libfoo.so dir3/libfoo.so
abidiff dir1/libfoo.so dir4/libfoo.so

without having to read dir1/libfoo.so three times and the other would be more
like:

abidiff --combinations dir1/libfoo.so dir2/libfoo.so dir3/libfoo.so
dir4/libfoo.so

abidiff dir1/libfoo.so dir2/libfoo.so
abidiff dir1/libfoo.so dir3/libfoo.so
abidiff dir1/libfoo.so dir4/libfoo.so
abidiff dir2/libfoo.so dir3/libfoo.so
abidiff dir2/libfoo.so dir4/libfoo.so
abidiff dir3/libfoo.so dir4/libfoo.so

and would only read and process dir[1-3]/libfoo.so one time each rather than
three times each.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19081] Make abipkgdiff be multi-threaded

[ooprala at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19081

Ondrej Oprala <ooprala at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
                 CC|                            |ooprala at redhat dot com
           Assignee|dodji at redhat dot com            |ooprala at redhat dot com

-- 
You are receiving this mail because:
You are on the CC list for the bug.