[Bug default/18838] Normalize the output of abidw

[Bug default/18838] Normalize the output of abidw

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

--- Comment #5 from dodji at redhat dot com ---
(In reply to andrew.c.morrow from comment #4)

> Needing to have access to two versions of libb.so is also problematic in the
> context of a build system like SCons, because there typically is no way to
> retain the prior version of libb.so, or name such a thing in the dependency
> graph.

Ah, I see.

There is a --weak-mode of abicompat, with which you just need to have access to
the later version of libb.so.  i.e:

  abicompat --weak-mode liba.so libb.so

But in that case, abicompat won't tell you if a symbol which liba.so consumes
from libb.so disappears from libb.so.  It'll just tell you if the types
consumed by liba.so from libb.so stop meaning the "same thing".

So close, and yet so far.  So yeah, I agree with you. 

There is another thing to keep in mind.  Right now, even if two versions of
libb.so have the *same* ABI, their corresponding abidw output might be
different.

This is mainly because the output of abidw is not normalized.  Hence this
enhancement request.  Also, even when the output of abidw becomes normalized, a
new version of libabigail might make that that output slightly change, for
instance, because we add support for new ABI artifacts.

But then, that is just a missed optimization, I agree with you. For the sake of
the intellectual correctness, I thought I'd mention it nonetheless.

Thank you for following up on this.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] New: Normalize the output of abidw

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

            Bug ID: 18838
           Summary: Normalize the output of abidw
           Product: libabigail
           Version: unspecified
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: default
          Assignee: dodji at redhat dot com
          Reporter: dodji at redhat dot com
                CC: libabigail at sourceware dot org
  Target Milestone: ---

Normalize the native XML representation of ABIs that libabigail emits (what
abidw emits, basically) so that two binaries with the same ABI always (mostly)
end-up having the same ABI XML.

This means basically reasonably sorting decls in their contexts, normalizing
spaces (okay this is already done by the XML emitted), and things like that.

Also, maybe we should start emitting versions of the ABI XML format ...

This is important for upstream projects willing to store ABI XML as ABI
baselines, so that they can review baseline changes by looking at diffs of the
XML.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[andrew.c.morrow at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

--- Comment #4 from andrew.c.morrow at gmail dot com ---
I agree with what you are saying in the sense of deciding manually if an
application should re-link. Not all ABI changes in a dependent library require
a re-link.

However, my use case is more narrow and less procedurally flexible.

I'm interested in the ABI hash as part of a best-effort elimination of needless
re-links in a build system that generates lots of shared libraries. In other
words, if liba.so depends on libb.so, and libb.so was just rebuilt, I'd like to
avoid rebuilding liba.so - if possible. If the ABI hash of libb.so didn't
change, then I definitely don't need to re-link liba.so. If it did change then
I'll just re-link liba.so. Perhaps that wasn't strictly necessary as you point
out, but at worst that is a missed build time optimization.

Needing to have access to two versions of libb.so is also problematic in the
context of a build system like SCons, because there typically is no way to
retain the prior version of libb.so, or name such a thing in the dependency
graph.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[andrew.c.morrow at gmail dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

andrew.c.morrow at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |andrew.c.morrow at gmail dot com

--- Comment #1 from andrew.c.morrow at gmail dot com ---
This is also important when using the output of abidw to prune unnecessary
dynamic relinks in a build system.

It might also be useful for that use case to have an 'abihash' utility, which
would traverse the libabigail data structures, but instead of emitting an XML
representation, would just emit a summary MD5 or SHA1 of the relevant info.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[dodji at seketeli dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

--- Comment #7 from dodji at seketeli dot org ---
Woops, I hit the wrong button too soon.  Here is what I wanted to say:

Another thing I think should be added to the xml output (I am calling the xml
format abixml now) is annotations to make the output easier to read and
understand.

for instance, if in abixml you have this:

   <type-decl name='char' size-in-bits='8' id='type-id-7'/>
    (...)
   <qualified-type-def type-id='type-id-7' const='yes' id='type-id-8'/>
    (...)
   <pointer-type-def type-id='type-id-8' id='type-id-9'/>

The second and third lines becomes easier to read when accompanied by a comment
like:

   <type-decl name='char' size-in-bits='8' id='type-id-7'/>
    (...)
   <!-- const char -->
   <qualified-type-def type-id='type-id-7' const='yes' id='type-id-8'/>
    (...)
   <!-- const char* -->
   <pointer-type-def type-id="type-id-8' id='type-id-8'/>

This eases reviews of changes to abixml files.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

--- Comment #2 from dodji at redhat dot com ---
(In reply to andrew.c.morrow from comment #1)
> This is also important when using the output of abidw to prune unnecessary
> dynamic relinks in a build system.
> 
> It might also be useful for that use case to have an 'abihash' utility,
> which would traverse the libabigail data structures, but instead of emitting
> an XML representation, would just emit a summary MD5 or SHA1 of the relevant
> info.

I have been thinking about this.

I think that if comparing two versions of a library L is fast enough, it might
be better to do that, and if the ABI changed in an incompatible way, then
trigger the re-linking of applications that linked against the older version of
L.

I am saying this because I think that having the ABIs of the two versions of L
being different doesn't necessary imply that an application using L should be
re-linked against the newer version of L. For instance, if a new entry point
got added to L and the application doesn't use that new entry point, then it
doesn't need to be re-linked.  Heck, even if an entry got *removed* from L and
the application doesn't use that entry point, no re-linking should be
necessary.

This is actually what the tool 'abicompat' is for.  If it detects that the
application is impacted by an ABI change that happened in L, then the
application ought to be re-linked.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[dodji at seketeli dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

dodji at seketeli dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dodji at seketeli dot org

--- Comment #6 from dodji at seketeli dot org ---
Another thing I think should be added to the xml output (I am calling the xml
format abixml now) is annotations to make the output easier to read and
understand.

for instance, if in abixml you have this:

   <type-decl name='char' size-in-bits='8' id='type-id-7'/>
    (...)
   <qualified-type-def type-id='type-id-7' const='yes' id='type-id-8'/>

The second line becomes easier to read if it's accompanied by a comment like:

   <type-decl name='char' size-in-bits='8' id='type-id-7'/>
    (...)

   <qualified-type-def type-id='type-id-7' const='yes' id='type-id-8'/>

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

--- Comment #3 from dodji at redhat dot com ---
(In reply to dodji from comment #2)

[...]

> This is actually what the tool 'abicompat' is for.  If it detects that the
> application is impacted by an ABI change that happened in L, then the
> application ought to be re-linked.

I forgot to mention that abicompat is documented at
https://sourceware.org/libabigail/manual/abicompat.html.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[roland at gnu dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

Roland McGrath <roland at gnu dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |19843


Referenced Bugs:

https://sourceware.org/bugzilla/show_bug.cgi?id=19843
[Bug 19843] make abidw usable in the glibc build process
-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[roland at gnu dot org]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

Roland McGrath <roland at gnu dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |roland at gnu dot org

--- Comment #8 from Roland McGrath <roland at gnu dot org> ---
This is another thing we'll need for glibc to start using abidw as part of its
build process.  We want abixml files we can commit to the source tree as
reference, and changes get manually reviewed by humans.  So normalization is
useful to make changes "patch-friendly", i.e. such that the patch between the
normal abixml of libfoo v1 and the normal abixml of libfoo v2 is easily
recognized by a human as "just adding the new stuff".

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug default/18838] Normalize the output of abidw

[dodji at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=18838

dodji at redhat dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #9 from dodji at redhat dot com ---
These days, the abixml format is "quite" normalized.

That means, a given version of libabigail will always emit the same abixml
output for the same binary.

This feature is used by libabigail itself in its regression test suite.

There are sabidw options to help with some useful cases.  For instance, one can
be willing to avoid emitting some absolute paths in some attributes of the
abixml documents, because those might depend on where the binary was built.

There is also the new --annotate option now that emits (human readable)
comments (aka annotations) describing the types and decls that are formally
defined in the abixml output.  

The idea is to help humans decipher the abixml output, and also, whenever some
"details" of the abixml change without changing the overall meaning of a type
or decl, we expect the annotation to stay the same.

So I guess this bug can now be closed.

Thank you for filling this!

-- 
You are receiving this mail because:
You are on the CC list for the bug.