[Bug default/19638] New: Different output for abidiff a.xml b.xml and abidiff a.xml b.so

[Bug default/19638] New: Different output for abidiff a.xml b.xml and abidiff a.xml b.so

[michi.henning at canonical dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

            Bug ID: 19638
           Summary: Different output for abidiff a.xml b.xml and abidiff
                    a.xml b.so
           Product: libabigail
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: default
          Assignee: dodji at redhat dot com
          Reporter: michi.henning at canonical dot com
                CC: libabigail at sourceware dot org
  Target Milestone: ---

I have a base ABI dump in libunity-scopes_1.0.3.abi.xml and a newly-built
library in libunity-scopes.so.1.0.3.

Now I create an ABI dump for the 1.0.3 version like so:

$ abidw --out-file libunity-scopes_1.0.3.abi.xml libunity-scopes.so.1.0.3

And here is a comparison of the base dump against the .so:

$ abidiff --no-unreferenced-symbols --suppressions suppressions
libunity-scopes_1.0.0.abi.xml libunity-scopes_1.0.3.abi.xml 
Functions changes summary: 0 Removed, 0 Changed, 0 Added function (160 filtered
out)
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

This is what I expect because the two versions are indeed compatible.

Now compare the ABIs again, this time using the base ABI dump and the .so file.
(I have truncated the output to save space.)

$ abidiff --no-unreferenced-symbols --suppressions suppressions
libunity-scopes_1.0.0.abi.xml libunity-scopes.so.1.0.3 
Functions changes summary: 0 Removed, 5 Changed, 1 Added functions (178
filtered out)
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

1 Added function:

  'method bool
unity::scopes::internal::MiddlewareFactory::MiddlewareData::operator<(const
unity::scopes::internal::MiddlewareFactory::MiddlewareData&)'   
{_ZNK5unity6scopes8internal17MiddlewareFactory14MiddlewareDataltERKS3_}

5 functions with some indirect sub-type change:

  [C]'method unity::scopes::ConfigException::ConfigException(int, const
unity::scopes::ConfigException&)' at ScopeExceptions.cpp:102:1 has some
indirect sub-type changes:
    linkage names of method
unity::scopes::ConfigException::ConfigException(int, const
unity::scopes::ConfigException&)
    changed from
'_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE,
_ZN5unity6scopes15ConfigExceptionC2ERKS1_' to
'_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'
    parameter 2 of type 'const unity::scopes::ConfigException&' changed:
      in referenced type 'const unity::scopes::ConfigException':
        'const unity::scopes::ConfigException' changed to 'const
std::__cxx11::string'

This is rather surprising. Not that, instead of 160 suppressed added functions,
this now reports 178 suppressed added functions. It also shows an additional
function in a namespace that isn't suppressed. Why the difference between the
dump file and the .so?

But the serious problem is that, in the second run, I also 5 incompatibilities
that are not reported by the first one.

I would expect it to make no difference when I run

abidiff lib0.xml lib3.xml

as opposed to

abidiff lib0.xml lib3.so

The output should always be identical, I would have thought?

Note that the issue seems to be related to alias entries in the base dump. For
example:

<elf-symbol
name='_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'
type='func-type' binding='global-binding'
alias='_ZN5unity6scopes15ConfigExceptionC2ERKS1_' is-defined='yes'/>

If I delete this line from the base dump, I get 4 ABI conflicts instead of 5,
and 2 added functions instead of 1.

This also relates to the compile flags, it seems. The base dump was produced
from a .so compiled with -g -O2. The current lib was compiled with -g (without
-O2).

If I repeat the exercise with the current lib also compiled with -g -O2, the
output is clean again.

Is there anything abigail can do to deal with this? The problem for us is that,
when testing locally, people typically don't compile with optimization whereas
the builds that are run in our continuous integration machinery do use -O2.

It's really awkward if we get false positives locally, depending on what flags
are used.

When I compile with --coverage, I also get hundreds of added functions.

Any suggestions for how to deal with this? Basically, the goal here is to be
able to build locally and just go "make test" and have the ABI compliance check
pass.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] Different output for abidiff a.xml b.xml and abidiff a.xml b.so

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

--- Comment #4 from dodji at redhat dot com ---
OK, I think I understand what is happening with respect to the "added
functions" discrepancy.  I think it's Libabigail that "forgets" how some
function representation (provided by DWARF) relate to the address of some
symbols (provided by the ELF symbol table).  This loss of information happens
at type canonicalization time.  I think I probably have a fix for this.

Now I am looking at the discrepancy with respected to the "changed functions"
section.

So, in the .so file that was build with -O2 -g (the one you provided the base
abi dump for), a compiler optimization resulted in these two functions to have
their function symbols be aliases:

unity::scopes::ConfigException::ConfigException(const std::string&)

and 

unity::scopes::ConfigException::ConfigException(const
unity::scopes::ConfigException&)


Their respective symbols are
_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

and 

_ZN5unity6scopes15ConfigExceptionC2ERKS1_.

The fact that these symbols are aliases is represented in the abi dump by the
following xml element:

    <elf-symbol
name='_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'
type='func-type' binding='global-binding'
alias='_ZN5unity6scopes15ConfigExceptionC2ERKS1_' is-defined='yes'/>

I believe Libabigail is thus being tricked by how DWARF represents the copy
constructor ConfigException(const ConfigException&), in light of this
optimization:

Here is what libabigail sees from DWARF (and ELF):


<function-decl name='ConfigException'
mangled-name='_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'
filepath='/build/unity-scopes-api-CnWggJ/unity-scopes-api-1.0.3+16.04.20160210.5/include/unity/scopes/ScopeExceptions.h'
line='140' column='1' visibility='default' binding='global' size-in-bits='64'
elf-symbol-id='_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'>
[...]

See how the mangled name of the copy constructor is said to be
_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE.
 It should be 
_ZN5unity6scopes15ConfigExceptionC2ERKS1_, but then, as these two symbols are
the same, well, the compiler is right to emit debug info that says this!

But then, as a result, libabigail is comparing the function
ConfigException(const ConfigException&) from the old library, with the function
ConfigException(const string&) from the new library.  And it's (rightfully)
saying that parameter 'const ConfigException&' was changed to parameter 'const
string&' !

Why?

Because, the symbol of ConfigException(const ConfigException&) in the old
library is
_ZN5unity6scopes15ConfigExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
(an alias of _ZN5unity6scopes15ConfigExceptionC2ERKS1_) and the symbol of
ConfigException(const string&) in the new library has the same symbol!

So, libabigail does its job of also comparing the *types* of these symbols. 
And indeed, the ABI of the first build is different from the ABI of the second
build, though, not in an incompatible way.

Now to understand the reason why comparing the two ABI dumps is different from
comparing one ABI dump against the later DSO, I think I would need to get my
hand on the .so file compiled with -g -O2.  It doesn't have to be exactly the
same as the one you provided here, though.  It "just" have to contain the code
of the ConfigException constructors which exhibits the compiler optimization I
talked about.

Thanks.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] DWARF reader fails to link clone function to its root declaration

[dodji at seketeli dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

dodji at seketeli dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|DWARF reader fails to link  |DWARF reader fails to link
                   |cloned function to its root |clone function to its root
                   |declaration                 |declaration

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] Different output for abidiff a.xml b.xml and abidiff a.xml b.so

[michi.henning at canonical dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

--- Comment #6 from Michi Henning <michi.henning at canonical dot com> ---
The implementation of the derived exception methods looks like this:

ConfigException::ConfigException(std::string const& reason) :
    Exception("unity::scopes::ConfigException", reason)
{
}

ConfigException::ConfigException(ConfigException const&) = default;

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] Different output for abidiff a.xml b.xml and abidiff a.xml b.so

[michi.henning at canonical dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

--- Comment #3 from Michi Henning <michi.henning at canonical dot com> ---
I've thrown away the build env from which the base lib was created. If you need
the base .so, I can recreate a new test case from scratch. (It's just that it
takes an hour or two.)

Please let me know if you need this, and I'll be happy to re-create the test
case, including the base .so.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] Different output for abidiff a.xml b.xml and abidiff a.xml b.so

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

dodji at redhat dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #2 from dodji at redhat dot com ---
I have started to look into this.

I got the tarball with the content to reproduce the issue, thanks!

I would also need to have the binary which you use to emit the dump
scopes_1.0.0.abi.xml.  Would it be possible that I download it from somewhere
as well?

Thanks.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] DWARF reader fails to link cloned function to its root declaration

[dodji at seketeli dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

dodji at seketeli dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dodji at seketeli dot org
            Summary|Different output for        |DWARF reader fails to link
                   |abidiff a.xml b.xml and     |cloned function to its root
                   |abidiff a.xml b.so          |declaration

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] Different output for abidiff a.xml b.xml and abidiff a.xml b.so

[michi.henning at canonical dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

--- Comment #1 from Michi Henning <michi.henning at canonical dot com> ---
The files were too large to attach, so I've uploaded here.

Please grab a copy because this file won't be around forever.

https://www.dropbox.com/s/s2037v1hd353ph6/files.tar.gz?dl=0

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] DWARF reader fails to link clone function to its root declaration

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

dodji at redhat dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #7 from dodji at redhat dot com ---
This issue should hopefully be fixed by commit
https://sourceware.org/git/gitweb.cgi?p=libabigail.git;a=commit;h=1bb3461d1df2dfc76b3fcf5876e472732745b3cc
from the master branch of the git repository.

I believe it has an extensive explanation of the issues at hand and how I
propose to solve them.

The patch requires that you re-generate the base ABI dump
(libunity-scopes_1.0.0.abi.xml) because it was basically lacking some important
information.  Sorry about that.

Here is the output I am getting:

$ time abidiff --no-unreferenced-symbols --changed-fns
libunity-scopes.so.1.0.0.abi libunity-scopes.so.1.0.3.abi
Functions changes summary: 0 Removed, 0 Changed, 0 Added function (179 filtered
out)
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

$ time abidiff --no-unreferenced-symbols --changed-fns
libunity-scopes.so.1.0.0.abi libunity-scopes.so.1.0.3
Functions changes summary: 0 Removed, 0 Changed, 0 Added function (179 filtered
out)
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

$ 

libunity-scopes.so.1.0.0.abi has been generated from the
libunity-scopes.so.1.0.0 binary, compiled with -g -O2, using abidw.

libunity-scopes.so.1.0.3.abi has been generated from the
libunity-scopes.so.1.0.3 binary, compiled with -g, using abidw.

Thanks for your time and assistance on this issue, and sorry for the
inconvenience!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/19638] Different output for abidiff a.xml b.xml and abidiff a.xml b.so

[michi.henning at canonical dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=19638

--- Comment #5 from Michi Henning <michi.henning at canonical dot com> ---
OK, a very similar base .so is here:
https://www.dropbox.com/s/7qzwkrw9ammi22x/libunity-scopes.so.1.0.0?dl=0

It's not completely identical, but identical enough for the purpose of this
exercise. None of the functions that are flagged as incompatible have changed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.