* Re: idea: abigail abixml archive
2023-11-15 15:53 idea: abigail abixml archive Frank Ch. Eigler
@ 2023-11-17 12:59 ` Dodji Seketeli
2023-11-17 14:53 ` Frank Ch. Eigler
2023-11-27 19:17 ` Ben Woodard
2023-11-21 14:54 ` Giuliano Procida
2023-11-27 19:09 ` Ben Woodard
2 siblings, 2 replies; 9+ messages in thread
From: Dodji Seketeli @ 2023-11-17 12:59 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: libabigail, woodard
Hello,
"Frank Ch. Eigler" <fche@redhat.com> a écrit:
> Hi -
>
> I'd love some feedback about the following idea, related to using
> libabigail to assemble a crowdsourced database of abixml files for
> linux distros.
>
> The germ of the idea is that developers may need to know whether a
> binary they built or found is likely to be abi-compatible with a given
> distro / version.
Yes, if my memory serves, Ben Woodard (in copy of this message) was the
first one I heard talking about this feature.
I am guessing this is useful for users who build a binary on a distro
and would like to know if they can run it on another distro without
using things like containers.
Am I right in expressing the user need here or am I missing something?
> This is possible today by downloading the target distro binaries and
> running libabigail locally against them, or using front-end scripts
> like fedabipkgdiff that do the downloading first.
Today, even if you have access to the target distro, it's not practical
with the current tools to know if the binary is "ABI compatible" with
it.
If I understand things correctly, you would need to:
1/ Get the ABI of the (transitive closure) set of dependencies of
the binary on its original distro. I call it orig-deps-abi.
2/ Get the ABI of the set of dependencies of the binary on the
target distro. I call it target-deps-abi.
3/ compare target-deps-abi against orig-deps-abi.
abicompat, for instance, hasn't been designed for that. We would need
to either extend it to support 1-3 or come up with a new tool for that.
> But this is a pain if one wants to compare against a range of versions
> or foreign distros.
Indeed.
> So the idea is instead to let people use an public archive of abixml
> artifacts instead of the binaries. The abixml files are relatively
> tiny, barely-ever changing, and should be an effective proxy for the
> real binaries. It's just a small matter of (a) storing, (b)
> using, and (c) collecting it.
OK.
And so the tool that does [1-3] would just download get the archive of
source and target distributions and do the work locally. Correct?
> --------------------
>
> For storing this data, I envision overloading the libabigil git repo
> (or a new one) with storage of the abixml documents.
I am guessing the new tool itself would be yet another libabigail tool
alongside the 6 we already have, so it would be in the libabigail git
repo.
But the distro ABI archives would better be hosted in another git repo
somewhere. And users might have their own private distro ABI repo
somewhere as they see fit. WDYT?
Heck, it could even just be in a directory tree served by whatever
transport protocol users would see fit. Git would be our preferred
choice, of course. Similarly to the way distro packages themselves are
organized today.
> To keep it dead simple, there could be one branch per /etc/os-release
> $ID/$VERSION_ID, one file per shared library in the distribution. For
> example, a fedora-39-x86-64 copy of /usr/lib64/libc.so.6, the file
> abidw produces could sit at
>
> repo git://sourceware.org/git/libabigail.git
> branch gitabixml/fedora/39/x86_64
> file /usr/lib64/libc.so.6.xml
I would even go further as to put the binary inside a subdirectory tree
with all that information, making it somewhat independent from being in git:
> repo git://sourceware.org/git/distributions-abi.git
> file /fedora/39/x86_64/glibc/usr/lib64/libc.so.6.xml
Storing information about the package would be useful, for instance, to
handle conflicting packages that might have binaries with the same
path.
> (Symlinks in the distro fs could be represented as symlinks in git.)
ACK.
> Updates to the distro package of course happen. It seems natural to
> update the abixml file for the affected file(s) right there in place.
Yes. Updating an abixml file would just be an overwrite. We would not
need to handle merges etc.
> Since it may sometimes be desirable to track what package version
> (e.g. rpm n-v-r) is associated with the abixml data of a given
> version, we could use stylized records in the git commit text (or a
> git note, or maybe a tag). That would mean one git commit per updated
> package, with metadata message like:
>
> Package: glibc-2.38-7.fc39.x86_64
>
> Maybe abidw version tags would be useful to add.
I am not sure what you mean by abidw version tag.
A possible way to handle this in a way that is not dependent on Git
would be to store the originating n-v-r in the abixml directly. The
tool that emits the abixml (from the original package) would be able to
do that.
> --------------------
>
> For using this data, I envision abidiff / abicompat taking a new form
> for its right operand. It could be a git url identifying the distro
> branch or tag. libabigail would fetch the corresponding file.xml
> within that. Simplify/default the heck out of it for ease of use:
>
> export $BRANCH=fedora/39/x86_64
> abicompat /bin/myprogram gitabixml:$BRANCH
>
> (Where "gitabixml:" could instruct the tool to look at the sourceware
> libabigail git / gitweb / cgit server. Let users specify different or
> private git servers via environment variables or something.
Yes, something like that. I guess the specific will depend on what we
end up settling on for the above (and below).
> --------------------
>
> For collecting this data, I envision writing some distro-specific
> scripts, kind of like fedabipkgdiff, being run by contributors or
> ourselves. One flavour could run in operational installed distros,
> doing the equivalent of
>
> find $PATHS -name '*.so.*' | while read lib; do
> # or filter with elfclassify
> package=`rpm -qf "$lib"`
> abidw "$lib" | (cd $gitrepo/`dirname $lib`; cat > "$lib.xml")
> (cd $gitrepo; git commit -m"Package: $package" "$lib.xml")
> done
>
> and rerun that occasionally as updates flow down from the distro.
> This could be done on a single beefy box running containers with
> different distros.
Yes, I like the idea of having something incremental like this.
There would probably be some tweaks added to abidw to let it add a
"version string" (the N-V-R mentioned earlier) to the abidw, but these
are details. Also, we need to add a mode to libabigail to let it expect
debuginfod to find the debuginfo because today, it expects the user to
provide the debug info location in cases where it's not already
installed on the system. Again, this is a detail but it's going to
matter as soon as we the rubber hits the road.
> Another flavour could be to take a set of RPM/etc. archives on a
> filesystem (or an ISO image), incrementally decompress them, run abidw
> on the individual files, and similarly construct the git repo of
> abixml files. (This is kind of like how debuginfod produces indexes
> from a bunch of RPMs.)
ACK.
>
> No matter how the local git repo is populated, each branch describing
> a data contributor's distro could be pushed to the central one,
> bringing that one up to date. Patches representing updates could be
> emailed too, but no one will want to read/review that stuff. We'd
> probably need a trusted pool of contributors who can just commit to
> areas of the central git repo. Secured with gitsigur of course. :-)
>
> The central repo could be built up entirely gradually. If some
> libraries were omitted from initial commits for a distro, a later
> contribution could fill in the gaps.
ACK.
Thank you for putting this thoughts together. The idea is getting much
less abstract in my mind now.
Cheers,
--
Dodji
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: idea: abigail abixml archive
2023-11-17 12:59 ` Dodji Seketeli
@ 2023-11-17 14:53 ` Frank Ch. Eigler
2023-11-17 23:06 ` Dodji Seketeli
2023-11-27 19:17 ` Ben Woodard
1 sibling, 1 reply; 9+ messages in thread
From: Frank Ch. Eigler @ 2023-11-17 14:53 UTC (permalink / raw)
To: Dodji Seketeli; +Cc: libabigail, woodard
Dodji Seketeli <dodji@seketeli.org> writes:
>> The germ of the idea is that developers may need to know whether a
>> binary they built or found is likely to be abi-compatible with a given
>> distro / version.
>
> Yes, if my memory serves, Ben Woodard (in copy of this message) was the
> first one I heard talking about this feature.
Yes.
> I am guessing this is useful for users who build a binary on a distro
> and would like to know if they can run it on another distro without
> using things like containers.
Yes.
> Am I right in expressing the user need here or am I missing something?
And also to detect accidental ABI changes within one's own new binary or
shared library, as compared with previous versions. Probably other uses
too.
> Today, even if you have access to the target distro, it's not practical
> with the current tools to know if the binary is "ABI compatible" with
> it.
>
> If I understand things correctly, you would need to:
>
> 1/ Get the ABI of the (transitive closure) set of dependencies of
> the binary on its original distro. I call it orig-deps-abi.
> 2/ Get the ABI of the set of dependencies of the binary on the
> target distro. I call it target-deps-abi.
> 3/ compare target-deps-abi against orig-deps-abi.
Maybe? Is the transitive closure necessary? Heck, if we're just
testing binary A against the remote shared libraries B C D, then the
local libraries B C D may not be relevant at all. If so, then it's just
comparing the local binary A vs. the directly dependentent remote
libraries mentioned in DT_NEEDED.
(To also compare local libraries B C D, one might rerun the tool with
each of B C D as the left operand.)
>[...]
> OK.
>
> And so the tool that does [1-3] would just download get the archive of
> source and target distributions and do the work locally. Correct?
The target distributions, yes. Source distributions if relevant (not
sure), but then again for source distribution it already has or could
get the binaries+debuginfo (at the cost of downloading time).
>> For storing this data, I envision overloading the libabigil git repo
>> (or a new one) with storage of the abixml documents.
>
> I am guessing the new tool itself would be yet another libabigail tool
> alongside the 6 we already have, so it would be in the libabigail git
> repo.
Sure.
> But the distro ABI archives would better be hosted in another git repo
> somewhere. And users might have their own private distro ABI repo
> somewhere as they see fit. WDYT?
Any random git repo, sure.
> Heck, it could even just be in a directory tree served by whatever
> transport protocol users would see fit. Git would be our preferred
> choice, of course. Similarly to the way distro packages themselves are
> organized today.
Perhaps one could accept the file:// url configuration, and access it
directly as though it were a git working tree checkout. But:
>> To keep it dead simple, there could be one branch per /etc/os-release
>> $ID/$VERSION_ID, one file per shared library in the distribution. For
>> example, a fedora-39-x86-64 copy of /usr/lib64/libc.so.6, the file
>> abidw produces could sit at
>>
>> repo git://sourceware.org/git/libabigail.git
>> branch gitabixml/fedora/39/x86_64
>> file /usr/lib64/libc.so.6.xml
>
> I would even go further as to put the binary inside a subdirectory tree
> with all that information, making it somewhat independent from being in git:
hmm
>> repo git://sourceware.org/git/distributions-abi.git
>> file /fedora/39/x86_64/glibc/usr/lib64/libc.so.6.xml
^^^^^^^^^^^^^^^^^^#####
While the ^^^^ first part can be known to the client, the #### package
abbreviation may well not be. A binary just makes reference to sonames,
which are only roughly file names, and definitely not package
abbreviations.
> Storing information about the package would be useful, for instance, to
> handle conflicting packages that might have binaries with the same
> path.
In a way, that's a variant of having more than one version of the same
binary in the history of the same distro.
>>[...]
>> Maybe abidw version tags would be useful to add.
>
> I am not sure what you mean by abidw version tag.
(I meant the "abidw --version" string to identify the producer.)
> A possible way to handle this in a way that is not dependent on Git
> would be to store the originating n-v-r in the abixml directly. The
> tool that emits the abixml (from the original package) would be able to
> do that.
Right, that's possible. But compared to a dead filesystem
representation, git lets us store multiple objects with the same name,
and lets the client choose which one to use (by traversing a branch
history by time or metadata hint in the commit or the actual payload
abixml itself). In a dead filesystem, such alternative/history
operation would have to be done by materializing the history/variant
axis somewhere in the path ... and clients would have to glob over that
component. (Adding "glibc" wouldn't be enough. n-v-r-a at the least.)
>> [...]
>> and rerun that occasionally as updates flow down from the distro.
>> This could be done on a single beefy box running containers with
>> different distros.
>
> Yes, I like the idea of having something incremental like this.
Right.
> Also, we need to add a mode to libabigail to let it expect debuginfod
> to find the debuginfo because today, it expects the user to provide
> the debug info location in cases where it's not already installed on
> the system. [...]
Are you sure? I run libabigail tools without any manual -d type
settings or debuginfod overrides, and it already just works, via
elfutils' builtin debuginfod support.
- FChE
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: idea: abigail abixml archive
2023-11-17 14:53 ` Frank Ch. Eigler
@ 2023-11-17 23:06 ` Dodji Seketeli
2023-12-06 0:14 ` Frank Ch. Eigler
0 siblings, 1 reply; 9+ messages in thread
From: Dodji Seketeli @ 2023-11-17 23:06 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: libabigail, woodard
Hello,
[...]
Dodji Seketeli <dodji@seketeli.org> writes:
>> Today, even if you have access to the target distro, it's not practical
>> with the current tools to know if the binary is "ABI compatible" with
>> it.
>>
>> If I understand things correctly, you would need to:
>>
>> 1/ Get the ABI of the (transitive closure) set of dependencies of
>> the binary on its original distro. I call it orig-deps-abi.
>> 2/ Get the ABI of the set of dependencies of the binary on the
>> target distro. I call it target-deps-abi.
>> 3/ compare target-deps-abi against orig-deps-abi.
fche@redhat.com (Frank Ch. Eigler) a écrit:
> Heck, if we're just testing binary A against the remote shared
> libraries B C D, then the local libraries B C D may not be relevant at
> all.
I think it all boils down to what we mean by "testing binary A against
its dependency".
When we want to know if a binary A is compatible with a library B, we
consider functions & global variables undefined in A and defined (and
exported) in B. The DWARF for the types of the undefined functions &
variables are present in A. These are the types "expected" by A. In B,
there is going to be same types, because they come with the definition
of the exported functions & variables that are expected by A. So these
types are "provided" by B.
We can compare the types expected by A to their counterpart provided by
B and if they are equal, then A is compatible with B, as far as these
functions & variables and their types are concerned.
This is basically what abicompat does.
But then there is a case of incompatibility that we can't catch by
looking at just A and B. That is, if a function that was previously
defined in B (and is needed by A) is removed from B. In that case, we
need to know if that function was moved to another dependency of A. It
only is if the function is removed from all the dependencies of A that
we can know for sure that the function is missing, and thus, that A is
facing an incompatible change in that regard.
> Is the transitive closure necessary?
I think the transitive closure would be necessary for the case I
explained above, because a removed function could have been moved to any
other dependant library. Or am I missing something?
> If so, then it's just comparing the local binary A vs. the
> directly dependentent remote libraries mentioned in DT_NEEDED.
If you only want to consider the dependant remote dependencies, then
that would amount to doing what abicompat does today, more or less. I
think that could work too (instead of going through [1-3]). The only
caveat I would have is the transitive closure one.
[...]
fche@redhat.com (Frank Ch. Eigler) a écrit:
>>> To keep it dead simple, there could be one branch per /etc/os-release
>>> $ID/$VERSION_ID, one file per shared library in the distribution. For
>>> example, a fedora-39-x86-64 copy of /usr/lib64/libc.so.6, the file
>>> abidw produces could sit at
>>>
>>> repo git://sourceware.org/git/libabigail.git
>>> branch gitabixml/fedora/39/x86_64
>>> file /usr/lib64/libc.so.6.xml
Dodji Seketeli <dodji@seketeli.org> writes:
>>
>> I would even go further as to put the binary inside a subdirectory tree
>> with all that information, making it somewhat independent from being in git:
fche@redhat.com (Frank Ch. Eigler) a écrit:
> hmm
>>> repo git://sourceware.org/git/distributions-abi.git
>>> file /fedora/39/x86_64/glibc/usr/lib64/libc.so.6.xml
> ^^^^^^^^^^^^^^^^^^#####
>
> While the ^^^^ first part can be known to the client, the #### package
> abbreviation may well not be. A binary just makes reference to sonames,
> which are only roughly file names, and definitely not package
> abbreviations.
Indeed, not all binary present on the user's system come from a package.
And the ABI of those binaries might indeed end up in the repository.
Point taken.
OK then, I take the package thing back.
Let's just store the thing in Git as you proposed.
[...]
>> Also, we need to add a mode to libabigail to let it expect debuginfod
>> to find the debuginfo because today, it expects the user to provide
>> the debug info location in cases where it's not already installed on
>> the system. [...]
>
> Are you sure? I run libabigail tools without any manual -d type
> settings or debuginfod overrides, and it already just works, via
> elfutils' builtin debuginfod support.
>
Interesting. I must double check. I must have broken something locally
then. I am pretty sure to remember I've had to add the --d{1,2} to
abipkgdiff otherwise it wasn't finding debuginfo. But I need to double
check. It it works that's great :-)
Cheers,
--
Dodji
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: idea: abigail abixml archive
2023-11-17 23:06 ` Dodji Seketeli
@ 2023-12-06 0:14 ` Frank Ch. Eigler
0 siblings, 0 replies; 9+ messages in thread
From: Frank Ch. Eigler @ 2023-12-06 0:14 UTC (permalink / raw)
To: Dodji Seketeli; +Cc: libabigail, woodard
Hi -
Having gone quiet for a few weeks, would like to show a really early
sketch of this gadget. See the tools/abidb python script in the
users/fche/abidb branch. I am better educated about limitations of
abicompat as discussed downthread, but this script doesn't attempt to
solve the whole problem yet. Recursive searches for SOLIB chains,
reverse dependencies for weak symbols etc., are not in there, but
some of them could be added. Anyway, what there is now:
% abidb --help
usage: abidb [-h] [--loglevel LOGLEVEL] [--git GIT]
[--distrobranch DISTROBRANCH] [--submit [SUBMIT ...]]
[--check [CHECK ...]] [--ld-library-path LD_LIBRARY_PATH]
Check binary against abidb corpus and/or submit new data.
options:
-h, --help show this help message and exit
--loglevel LOGLEVEL logging level (default: info)
--git GIT abidb git working tree (default: .)
--distrobranch DISTROBRANCH
use given abidb distrobranch (default:
fedora/39/x86_64)
--submit [SUBMIT ...]
submit abidw of given binaries to abidb (default: [])
--check [CHECK ...] check given binaries against abidb (default: [])
--ld-library-path LD_LIBRARY_PATH
LD_LIBRARY_PATH to use for soname resolution during
check (default: /lib:/lib64)
% git init /tmp/abidb
NB: this git repo would naturally be centrally shareable over time.
% abidb --git /tmp/abidb --submit /lib*/libc.so.?
2023-12-05 19:07:58,738:abidb:INFO:checked out distrobranch fedora/39/x86_64 files 0
2023-12-05 19:07:59,268:abidb:INFO:binary /lib64/libc.so.6 abidb-path /tmp/abidb/lib64/libc.so.6/7dd93cb9991a89f0ec53d9443a0b78ad952269bc.xml abixml-length 3305422 commit e90283c5fe45a2c465edfeb2ec55f3b47cf6dce3
2023-12-05 19:08:00,226:abidb:INFO:binary /lib/libc.so.6 abidb-path /tmp/abidb/lib/libc.so.6/02ac2b73b0b28086109ca73389bf97a26fd70542.xml abixml-length 3521035 commit 57536c5b38e9463654432c0b31e4ff5401eed512
NB: the directory structure in git mirrors the filesystem, plus the
appropriate BUILDID. This allows multiple different updates of the
same library to be represented.
% abidb --git /tmp/abidb --check /bin/ls
2023-12-05 19:08:35,699:abidb:INFO:checked out distrobranch fedora/39/x86_64 files 2
2023-12-05 19:08:35,701:abidb:WARNING:no abi.xml found for libselinux.so.1 searching under /lib:/lib64
2023-12-05 19:08:35,701:abidb:WARNING:no abi.xml found for libcap.so.2 searching under /lib:/lib64
2023-12-05 19:08:35,867:abidb:INFO:abicompat success: /bin/ls vs. /tmp/abidb/lib/libc.so.6/02ac2b73b0b28086109ca73389bf97a26fd70542.xml
2023-12-05 19:08:36,022:abidb:INFO:abicompat success: /bin/ls vs. /tmp/abidb/lib64/libc.so.6/7dd93cb9991a89f0ec53d9443a0b78ad952269bc.xml
NB: it complains not having a few solibs in the database yet, so let's add them:
% abidb --git /tmp/abidb --submit /lib*/libselinux.so.? /lib*/libcap.so.?
2023-12-05 19:09:10,984:abidb:INFO:checked out distrobranch fedora/39/x86_64 files 2
2023-12-05 19:09:11,041:abidb:INFO:binary /lib64/libselinux.so.1 abidb-path /tmp/abidb/lib64/libselinux.so.1/c3ca2b00f1505a173d44417dc98dced2ec390268.xml abixml-length 281444 commit 91e92229bf17b00cb708d5f0da3e9d94708665a2
2023-12-05 19:09:11,087:abidb:INFO:binary /lib/libselinux.so.1 abidb-path /tmp/abidb/lib/libselinux.so.1/b4ea9035186f94416600ccf6b378afb408a8ec8e.xml abixml-length 280693 commit 9979a994ef5d77ab25ce777ab22ff10aae3d4f59
2023-12-05 19:09:11,119:abidb:INFO:binary /lib64/libcap.so.2 abidb-path /tmp/abidb/lib64/libcap.so.2/ff26c67e54bb86d066f1c3cee3cef74fe0c13a77.xml abixml-length 68942 commit 780a099d2a59132aca0f82ab6d5e553be08c2969
2023-12-05 19:09:11,144:abidb:INFO:binary /lib/libcap.so.2 abidb-path /tmp/abidb/lib/libcap.so.2/86ab6783bd93f43139db2924cbbf25bc9419dc9c.xml abixml-length 69053 commit 7f9cec82580c58007748f470408030195d9dcd07
NB: now it will be happy
% abidb --git /tmp/abidb --check /bin/ls
2023-12-05 19:09:27,561:abidb:INFO:checked out distrobranch fedora/39/x86_64 files 6
2023-12-05 19:09:27,578:abidb:INFO:abicompat success: /bin/ls vs. /tmp/abidb/lib/libselinux.so.1/b4ea9035186f94416600ccf6b378afb408a8ec8e.xml
2023-12-05 19:09:27,590:abidb:INFO:abicompat success: /bin/ls vs. /tmp/abidb/lib64/libselinux.so.1/c3ca2b00f1505a173d44417dc98dced2ec390268.xml
2023-12-05 19:09:27,596:abidb:INFO:abicompat success: /bin/ls vs. /tmp/abidb/lib/libcap.so.2/86ab6783bd93f43139db2924cbbf25bc9419dc9c.xml
2023-12-05 19:09:27,602:abidb:INFO:abicompat success: /bin/ls vs. /tmp/abidb/lib64/libcap.so.2/ff26c67e54bb86d066f1c3cee3cef74fe0c13a77.xml
2023-12-05 19:09:27,768:abidb:INFO:abicompat success: /bin/ls vs. /tmp/abidb/lib/libc.so.6/02ac2b73b0b28086109ca73389bf97a26fd70542.xml
2023-12-05 19:09:27,920:abidb:INFO:abicompat success: /bin/ls vs. /tmp/abidb/lib64/libc.so.6/7dd93cb9991a89f0ec53d9443a0b78ad952269bc.xml
NB: ok let's try some random ancient binary
% abidb --git /tmp/abidb --check /usr/local/bin/avplay
2023-12-05 19:11:17,151:abidb:INFO:checked out distrobranch fedora/39/x86_64 files 6
2023-12-05 19:11:17,153:abidb:WARNING:no abi.xml found for libm.so.6 searching under /lib:/lib64
2023-12-05 19:11:17,153:abidb:WARNING:no abi.xml found for libasound.so.2 searching under /lib:/lib64
2023-12-05 19:11:17,153:abidb:WARNING:no abi.xml found for libbz2.so.1 searching under /lib:/lib64
2023-12-05 19:11:17,153:abidb:WARNING:no abi.xml found for libz.so.1 searching under /lib:/lib64
2023-12-05 19:11:17,153:abidb:WARNING:no abi.xml found for libSDL-1.2.so.0 searching under /lib:/lib64
2023-12-05 19:11:17,153:abidb:WARNING:no abi.xml found for libpthread.so.0 searching under /lib:/lib64
2023-12-05 19:11:22,889:abidb:WARNING:abicompat error: /usr/local/bin/avplay vs. /tmp/abidb/lib/libc.so.6/02ac2b73b0b28086109ca73389bf97a26fd70542.xml:
functions defined in library '/tmp/abidb/lib/libc.so.6/02ac2b73b0b28086109ca73389bf97a26fd70542.xml'
have sub-types that are different from what application '/usr/local/bin/avplay' expects:
function int __xpg_strerror_r(int, char*, size_t):
parameter 3 of type 'typedef size_t' changed:
underlying type 'unsigned long int' changed:
type name changed from 'unsigned long int' to 'unsigned int'
type size changed from 64 to 32 (in bits)
address size of function changed from 64 bits to 32 bits
variables defined in library '/tmp/abidb/lib/libc.so.6/02ac2b73b0b28086109ca73389bf97a26fd70542.xml'
have sub-types that are different from what application '/usr/local/bin/avplay' expects:
void (void*, void*)* __free_hook:
in pointed to type 'function type void (void*, void*)':
address size of function changed from 64 bits to 32 bits
FILE* stdin:
in pointed to type 'typedef FILE' at FILE.h:7:1:
underlying type 'struct _IO_FILE' at struct_FILE.h:49:1 changed:
type size changed from 1728 to 1184 (in bits)
[...]
long int __timezone:
type size changed from 64 to 32 (in bits)
2023-12-05 19:11:22,889:abidb:INFO:abicompat success: /usr/local/bin/avplay vs. /tmp/abidb/lib/libc.so.6/02ac2b73b0b28086109ca73389bf97a26fd70542.xml
2023-12-05 19:11:28,727:abidb:WARNING:abicompat error: /usr/local/bin/avplay vs. /tmp/abidb/lib64/libc.so.6/7dd93cb9991a89f0ec53d9443a0b78ad952269bc.xml:
functions defined in library '/tmp/abidb/lib64/libc.so.6/7dd93cb9991a89f0ec53d9443a0b78ad952269bc.xml'
have sub-types that are different from what application '/usr/local/bin/avplay' expects:
function int fputc(int, FILE*):
parameter 2 of type 'FILE*' has sub-type changes:
in pointed to type 'typedef FILE' at FILE.h:7:1:
underlying type 'struct _IO_FILE' at struct_FILE.h:49:1 changed:
type size hasn't changed
[...]
unqualified underlying type 'struct tm' changed at struct_tm.h:7:1, as reported earlier
variables defined in library '/tmp/abidb/lib64/libc.so.6/7dd93cb9991a89f0ec53d9443a0b78ad952269bc.xml'
have sub-types that are different from what application '/usr/local/bin/avplay' expects:
FILE* stdin:
pointed to type 'typedef FILE' changed at FILE.h:7:1, as reported earlier
[...]
2023-12-05 19:11:28,727:abidb:INFO:abicompat success: /usr/local/bin/avplay vs. /tmp/abidb/lib64/libc.so.6/7dd93cb9991a89f0ec53d9443a0b78ad952269bc.xml
That's it for now.
- FChE
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: idea: abigail abixml archive
2023-11-17 12:59 ` Dodji Seketeli
2023-11-17 14:53 ` Frank Ch. Eigler
@ 2023-11-27 19:17 ` Ben Woodard
2023-11-28 13:52 ` Dodji Seketeli
1 sibling, 1 reply; 9+ messages in thread
From: Ben Woodard @ 2023-11-27 19:17 UTC (permalink / raw)
To: Dodji Seketeli, Frank Ch. Eigler; +Cc: libabigail
On 11/17/23 04:59, Dodji Seketeli wrote:
> Hello,
>
> "Frank Ch. Eigler" <fche@redhat.com> a écrit:
>
>> Hi -
>>
>> I'd love some feedback about the following idea, related to using
>> libabigail to assemble a crowdsourced database of abixml files for
>> linux distros.
>>
>> The germ of the idea is that developers may need to know whether a
>> binary they built or found is likely to be abi-compatible with a given
>> distro / version.
> Yes, if my memory serves, Ben Woodard (in copy of this message) was the
> first one I heard talking about this feature.
>
> I am guessing this is useful for users who build a binary on a distro
> and would like to know if they can run it on another distro without
> using things like containers.
>
> Am I right in expressing the user need here or am I missing something?
>
>> This is possible today by downloading the target distro binaries and
>> running libabigail locally against them, or using front-end scripts
>> like fedabipkgdiff that do the downloading first.
> Today, even if you have access to the target distro, it's not practical
> with the current tools to know if the binary is "ABI compatible" with
> it.
>
> If I understand things correctly, you would need to:
>
> 1/ Get the ABI of the (transitive closure) set of dependencies of
> the binary on its original distro. I call it orig-deps-abi.
>
> 2/ Get the ABI of the set of dependencies of the binary on the
> target distro. I call it target-deps-abi.
>
> 3/ compare target-deps-abi against orig-deps-abi.
>
> abicompat, for instance, hasn't been designed for that. We would need
> to either extend it to support 1-3 or come up with a new tool for that.
I continue to believe that the current design of abicompat is not
sufficient for what users want to do. The name is misleading and what it
currently accomplishes not only doesn't match user's expectations given
its name, What it does accomplish is not very useful from a user's
perspective. We have literally not been able to come up with an actual
use for abicompat as it now exists without putting a ton of scripting
around it. Therefore I strongly believe that it should be extended to
actually compute the forward and backward ABI dependencies of the
program and its library dependencies. This is what users expect
abicompat to do before they are educated on what it actually does and
this is what users need it to do.
>
>> But this is a pain if one wants to compare against a range of versions
>> or foreign distros.
> Indeed.
>
>> So the idea is instead to let people use an public archive of abixml
>> artifacts instead of the binaries. The abixml files are relatively
>> tiny, barely-ever changing, and should be an effective proxy for the
>> real binaries. It's just a small matter of (a) storing, (b)
>> using, and (c) collecting it.
> OK.
>
> And so the tool that does [1-3] would just download get the archive of
> source and target distributions and do the work locally. Correct?
>
>> --------------------
>>
>> For storing this data, I envision overloading the libabigil git repo
>> (or a new one) with storage of the abixml documents.
> I am guessing the new tool itself would be yet another libabigail tool
> alongside the 6 we already have, so it would be in the libabigail git
> repo.
>
> But the distro ABI archives would better be hosted in another git repo
> somewhere. And users might have their own private distro ABI repo
> somewhere as they see fit. WDYT?
>
> Heck, it could even just be in a directory tree served by whatever
> transport protocol users would see fit. Git would be our preferred
> choice, of course. Similarly to the way distro packages themselves are
> organized today.
>
>> To keep it dead simple, there could be one branch per /etc/os-release
>> $ID/$VERSION_ID, one file per shared library in the distribution. For
>> example, a fedora-39-x86-64 copy of /usr/lib64/libc.so.6, the file
>> abidw produces could sit at
>>
>> repo git://sourceware.org/git/libabigail.git
>> branch gitabixml/fedora/39/x86_64
>> file /usr/lib64/libc.so.6.xml
> I would even go further as to put the binary inside a subdirectory tree
> with all that information, making it somewhat independent from being in git:
>
>> repo git://sourceware.org/git/distributions-abi.git
>> file /fedora/39/x86_64/glibc/usr/lib64/libc.so.6.xml
>
> Storing information about the package would be useful, for instance, to
> handle conflicting packages that might have binaries with the same
> path.
>
>> (Symlinks in the distro fs could be represented as symlinks in git.)
> ACK.
>
>
>> Updates to the distro package of course happen. It seems natural to
>> update the abixml file for the affected file(s) right there in place.
> Yes. Updating an abixml file would just be an overwrite. We would not
> need to handle merges etc.
>
>> Since it may sometimes be desirable to track what package version
>> (e.g. rpm n-v-r) is associated with the abixml data of a given
>> version, we could use stylized records in the git commit text (or a
>> git note, or maybe a tag). That would mean one git commit per updated
>> package, with metadata message like:
>>
>> Package: glibc-2.38-7.fc39.x86_64
>>
>> Maybe abidw version tags would be useful to add.
> I am not sure what you mean by abidw version tag.
>
> A possible way to handle this in a way that is not dependent on Git
> would be to store the originating n-v-r in the abixml directly. The
> tool that emits the abixml (from the original package) would be able to
> do that.
>
>> --------------------
>>
>> For using this data, I envision abidiff / abicompat taking a new form
>> for its right operand. It could be a git url identifying the distro
>> branch or tag. libabigail would fetch the corresponding file.xml
>> within that. Simplify/default the heck out of it for ease of use:
>>
>> export $BRANCH=fedora/39/x86_64
>> abicompat /bin/myprogram gitabixml:$BRANCH
>>
>> (Where "gitabixml:" could instruct the tool to look at the sourceware
>> libabigail git / gitweb / cgit server. Let users specify different or
>> private git servers via environment variables or something.
> Yes, something like that. I guess the specific will depend on what we
> end up settling on for the above (and below).
>
>
>> --------------------
>>
>> For collecting this data, I envision writing some distro-specific
>> scripts, kind of like fedabipkgdiff, being run by contributors or
>> ourselves. One flavour could run in operational installed distros,
>> doing the equivalent of
>>
>> find $PATHS -name '*.so.*' | while read lib; do
>> # or filter with elfclassify
>> package=`rpm -qf "$lib"`
>> abidw "$lib" | (cd $gitrepo/`dirname $lib`; cat > "$lib.xml")
>> (cd $gitrepo; git commit -m"Package: $package" "$lib.xml")
>> done
>>
>> and rerun that occasionally as updates flow down from the distro.
>> This could be done on a single beefy box running containers with
>> different distros.
> Yes, I like the idea of having something incremental like this.
>
> There would probably be some tweaks added to abidw to let it add a
> "version string" (the N-V-R mentioned earlier) to the abidw, but these
> are details. Also, we need to add a mode to libabigail to let it expect
> debuginfod to find the debuginfo because today, it expects the user to
> provide the debug info location in cases where it's not already
> installed on the system. Again, this is a detail but it's going to
> matter as soon as we the rubber hits the road.
>
>> Another flavour could be to take a set of RPM/etc. archives on a
>> filesystem (or an ISO image), incrementally decompress them, run abidw
>> on the individual files, and similarly construct the git repo of
>> abixml files. (This is kind of like how debuginfod produces indexes
>> from a bunch of RPMs.)
> ACK.
>> No matter how the local git repo is populated, each branch describing
>> a data contributor's distro could be pushed to the central one,
>> bringing that one up to date. Patches representing updates could be
>> emailed too, but no one will want to read/review that stuff. We'd
>> probably need a trusted pool of contributors who can just commit to
>> areas of the central git repo. Secured with gitsigur of course. :-)
>>
>> The central repo could be built up entirely gradually. If some
>> libraries were omitted from initial commits for a distro, a later
>> contribution could fill in the gaps.
> ACK.
>
> Thank you for putting this thoughts together. The idea is getting much
> less abstract in my mind now.
>
> Cheers,
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: idea: abigail abixml archive
2023-11-27 19:17 ` Ben Woodard
@ 2023-11-28 13:52 ` Dodji Seketeli
0 siblings, 0 replies; 9+ messages in thread
From: Dodji Seketeli @ 2023-11-28 13:52 UTC (permalink / raw)
To: Ben Woodard; +Cc: Frank Ch. Eigler, libabigail
Ben Woodard <woodard@redhat.com> a écrit:
>> If I understand things correctly, you would need to:
>>
>> 1/ Get the ABI of the (transitive closure) set of dependencies of
>> the binary on its original distro. I call it orig-deps-abi.
>>
>> 2/ Get the ABI of the set of dependencies of the binary on the
>> target distro. I call it target-deps-abi.
>>
>> 3/ compare target-deps-abi against orig-deps-abi.
>>
>> abicompat, for instance, hasn't been designed for that. We would need
>> to either extend it to support 1-3 or come up with a new tool for that.
>
> I continue to believe that the current design of abicompat is not
> sufficient for what users want to do. The name is misleading and what
> it currently accomplishes not only doesn't match user's expectations
> given its name, What it does accomplish is not very useful from a
> user's perspective. We have literally not been able to come up with an
> actual use for abicompat as it now exists without putting a ton of
> scripting around it. Therefore I strongly believe that it should be
> extended to actually compute the forward and backward ABI dependencies
> of the program and its library dependencies. This is what users
> expect abicompat to do before they are educated on what it actually
> does and this is what users need it to do.
FWIW, as not everyone is familiar with the requests for enhancement
about abicompat, Ben has filed a number of these:
* https://sourceware.org/bugzilla/show_bug.cgi?id=27514
"recursively compare libraries in abicompat"
(This is somehow asking for 1/ above)
* https://sourceware.org/bugzilla/show_bug.cgi?id=27208
"abicompat doesn't consider references from the libraries to the main executable"
This one depends on:
* https://sourceware.org/bugzilla/show_bug.cgi?id=29008
"abicompat doesn't consider functions (or variables) whose symbol type is NOTYPE"
* https://sourceware.org/bugzilla/show_bug.cgi?id=29013
"abicompat doesn't consider weak symbol replacement"
[...]
Cheers,
--
Dodji
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: idea: abigail abixml archive
2023-11-15 15:53 idea: abigail abixml archive Frank Ch. Eigler
2023-11-17 12:59 ` Dodji Seketeli
@ 2023-11-21 14:54 ` Giuliano Procida
2023-11-27 19:09 ` Ben Woodard
2 siblings, 0 replies; 9+ messages in thread
From: Giuliano Procida @ 2023-11-21 14:54 UTC (permalink / raw)
To: Frank Ch. Eigler; +Cc: libabigail, Matthias Männich
Hi.
On Wed, 15 Nov 2023 at 15:53, Frank Ch. Eigler <fche@redhat.com> wrote:
>
> Hi -
>
> I'd love some feedback about the following idea, related to using
> libabigail to assemble a crowdsourced database of abixml files for
> linux distros.
>
> The germ of the idea is that developers may need to know whether a
> binary they built or found is likely to be abi-compatible with a given
> distro / version. This is possible today by downloading the target
> distro binaries and running libabigail locally against them, or using
> front-end scripts like fedabipkgdiff that do the downloading first.
> But this is a pain if one wants to compare against a range of versions
> or foreign distros.
>
> So the idea is instead to let people use an public archive of abixml
> artifacts instead of the binaries. The abixml files are relatively
> tiny, barely-ever changing, and should be an effective proxy for the
> real binaries. It's just a small matter of (a) storing, (b)
> using, and (c) collecting it.
>
> --------------------
>
> For storing this data, I envision overloading the libabigil git repo
> (or a new one) with storage of the abixml documents. To keep it dead
> simple, there could be one branch per /etc/os-release $ID/$VERSION_ID,
> one file per shared library in the distribution. For example, a
> fedora-39-x86-64 copy of /usr/lib64/libc.so.6, the file abidw produces
> could sit at
>
> repo git://sourceware.org/git/libabigail.git
> branch gitabixml/fedora/39/x86_64
> file /usr/lib64/libc.so.6.xml
>
> (Symlinks in the distro fs could be represented as symlinks in git.)
>
> Updates to the distro package of course happen. It seems natural to
> update the abixml file for the affected file(s) right there in place.
>
> Since it may sometimes be desirable to track what package version
> (e.g. rpm n-v-r) is associated with the abixml data of a given
> version, we could use stylized records in the git commit text (or a
> git note, or maybe a tag). That would mean one git commit per updated
> package, with metadata message like:
>
> Package: glibc-2.38-7.fc39.x86_64
>
> Maybe abidw version tags would be useful to add.
>
> --------------------
>
> For using this data, I envision abidiff / abicompat taking a new form
> for its right operand. It could be a git url identifying the distro
> branch or tag. libabigail would fetch the corresponding file.xml
> within that. Simplify/default the heck out of it for ease of use:
>
> export $BRANCH=fedora/39/x86_64
> abicompat /bin/myprogram gitabixml:$BRANCH
>
> (Where "gitabixml:" could instruct the tool to look at the sourceware
> libabigail git / gitweb / cgit server. Let users specify different or
> private git servers via environment variables or something.
>
> --------------------
>
> For collecting this data, I envision writing some distro-specific
> scripts, kind of like fedabipkgdiff, being run by contributors or
> ourselves. One flavour could run in operational installed distros,
> doing the equivalent of
>
> find $PATHS -name '*.so.*' | while read lib; do
> # or filter with elfclassify
> package=`rpm -qf "$lib"`
> abidw "$lib" | (cd $gitrepo/`dirname $lib`; cat > "$lib.xml")
> (cd $gitrepo; git commit -m"Package: $package" "$lib.xml")
> done
>
> and rerun that occasionally as updates flow down from the distro.
> This could be done on a single beefy box running containers with
> different distros.
>
> Another flavour could be to take a set of RPM/etc. archives on a
> filesystem (or an ISO image), incrementally decompress them, run abidw
> on the individual files, and similarly construct the git repo of
> abixml files. (This is kind of like how debuginfod produces indexes
> from a bunch of RPMs.)
>
> No matter how the local git repo is populated, each branch describing
> a data contributor's distro could be pushed to the central one,
> bringing that one up to date. Patches representing updates could be
> emailed too, but no one will want to read/review that stuff. We'd
> probably need a trusted pool of contributors who can just commit to
> areas of the central git repo. Secured with gitsigur of course. :-)
>
> The central repo could be built up entirely gradually. If some
> libraries were omitted from initial commits for a distro, a later
> contribution could fill in the gaps.
>
> --------------------
>
> OK, how reasonable does all this sound?
This sounds like an interesting project, but you can go further.
Starting at the level of a single binary with a single .so. The
questions we want answered are:
1. Will all undefined symbols resolve successfully (otherwise
reporting missing symbols)?
2. Are the types of the resolved symbols compatible (otherwise
reporting differences)?
3. Do we have ABI representations that let us answer 1. and 2. without
having some binaries to hand? (Not yet.)
Neither libabigail nor STG emit undefined symbols in their ABI
representations (yet), so answering 1. currently requires having
binaries and debug information to hand.
Now build this up to multiple binaries, SONAME, bundled and unbundled
shared objects, library dependencies (ELF needs), multiple
distributions, packages, versions and architectures, supporting
link-loaded plugins with dlopen etc.
Having a full database of existing libraries would allow compatibility
for a freshly-compiled binary (or package) to be checked (idea due to
Matthias Männich) or dependency hell to be explored without actually
installing any packages.
Similarly, if binaries (and their packages) are in the database,
questions about library upgrades could be answered.
Giuliano.
>
> - FChE
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: idea: abigail abixml archive
2023-11-15 15:53 idea: abigail abixml archive Frank Ch. Eigler
2023-11-17 12:59 ` Dodji Seketeli
2023-11-21 14:54 ` Giuliano Procida
@ 2023-11-27 19:09 ` Ben Woodard
2 siblings, 0 replies; 9+ messages in thread
From: Ben Woodard @ 2023-11-27 19:09 UTC (permalink / raw)
To: libabigail
[-- Attachment #1: Type: text/plain, Size: 7663 bytes --]
On 11/15/23 07:53, Frank Ch. Eigler wrote:
> Hi -
>
> I'd love some feedback about the following idea, related to using
> libabigail to assemble a crowdsourced database of abixml files for
> linux distros.
>
> The germ of the idea is that developers may need to know whether a
> binary they built or found is likely to be abi-compatible with a given
> distro / version. This is possible today by downloading the target
> distro binaries and running libabigail locally against them, or using
> front-end scripts like fedabipkgdiff that do the downloading first.
> But this is a pain if one wants to compare against a range of versions
> or foreign distros.
The other big use is "do I need to rebuild my binary for this new distro?"
This can be either a new Y release in RHEL X.Y where you have built for
RHEL 9.2 and you want to make damn well sure that your program is
compatible with RHEL9.3. Yes we make ABI stability guarantees but that
only covers a portion of the distro. Or it could be you have your
program built for Fedora 38 and you want to know if you need to rebuild
it for Fedora 39 or if it happens to be ABI compatible.
The other use case is within the distro. You are a packager and you just
applied a patch. You want to make sure that you didn't break ABI with
the patch before you release it onto the world.
>
> So the idea is instead to let people use an public archive of abixml
> artifacts instead of the binaries. The abixml files are relatively
> tiny, barely-ever changing, and should be an effective proxy for the
> real binaries. It's just a small matter of (a) storing, (b)
> using, and (c) collecting it.
a) generating
b) storing
c) using
d) collecting
I add a) generating because as I have shown in the past there are many
cases where libabigail needs to have some bugs fixed to be able to
generate abixml files for particular packages.
> --------------------
>
> For storing this data, I envision overloading the libabigil git repo
> (or a new one) with storage of the abixml documents. To keep it dead
> simple, there could be one branch per /etc/os-release $ID/$VERSION_ID,
> one file per shared library in the distribution. For example, a
> fedora-39-x86-64 copy of /usr/lib64/libc.so.6, the file abidw produces
> could sit at
>
> repo git://sourceware.org/git/libabigail.git
> branch gitabixml/fedora/39/x86_64
> file /usr/lib64/libc.so.6.xml
>
> (Symlinks in the distro fs could be represented as symlinks in git.)
>
> Updates to the distro package of course happen. It seems natural to
> update the abixml file for the affected file(s) right there in place.
>
> Since it may sometimes be desirable to track what package version
> (e.g. rpm n-v-r) is associated with the abixml data of a given
> version, we could use stylized records in the git commit text (or a
> git note, or maybe a tag). That would mean one git commit per updated
> package, with metadata message like:
>
> Package: glibc-2.38-7.fc39.x86_64
>
> Maybe abidw version tags would be useful to add.
Yeah, one challenge is going to be when for whatever reason dodji needs
to update the abixml file format. In theory it shouldn't happen that
often but when it does, it will be a HUGE churn in the git repo. I would
suggest to mitigate that problem, there should be a abixml version in
the pathname.
I know that one of the things that we need in the not too distant future
is to track ABI on inline functions in C++ and so I expect that will
need a AIBXML format change when it happens.
>
> --------------------
>
> For using this data, I envision abidiff / abicompat taking a new form
> for its right operand. It could be a git url identifying the distro
> branch or tag. libabigail would fetch the corresponding file.xml
> within that. Simplify/default the heck out of it for ease of use:
>
> export $BRANCH=fedora/39/x86_64
> abicompat /bin/myprogram gitabixml:$BRANCH
>
> (Where "gitabixml:" could instruct the tool to look at the sourceware
> libabigail git / gitweb / cgit server. Let users specify different or
> private git servers via environment variables or something.
As Dodji pointed out below abicompat doesn't currently handle
dependencies very well. This has been a long term disagreement between
us. To try to simplify (possibly oversimplify) the disagreement. Dodji
believes that abicompat does one thing and that is test the forward ABI
dependencies from the program to a particular library. Very Unixy do one
thing. I disagree with this philosophy and all of my discussions with
users indicate that they misinterpret what abicompat actually does
thinking that it does more than it does. I'm a strong proponent of
making abicompat do what the name implies and what users believe it does
and that is actually check that the library is compatible with the
program specified on the command line. This references another design
principle: The principle of least suprise.
Thus IMHO abicompat should:
1) examine both forward and back dependencies. i.e. if a library makes
use of a function or variable in the program itself then abicompat
should consider that an ABI mismatch.
2) it should include all the dependencies of the program when
considering ABI compatibility. Yes this could be made into a script but
it ends up being a very inefficient and complicated script. I've written
it two different times and people complain about the performance. The
problem is it has to load the abixml over and over rather than keeping
it in memory when doing the comparison. On complicated programs with
lots of dependencies, this takes considerable time.
>
> --------------------
>
> For collecting this data, I envision writing some distro-specific
> scripts, kind of like fedabipkgdiff, being run by contributors or
> ourselves. One flavour could run in operational installed distros,
> doing the equivalent of
>
> find $PATHS -name '*.so.*' | while read lib; do
> # or filter with elfclassify
> package=`rpm -qf "$lib"`
> abidw "$lib" | (cd $gitrepo/`dirname $lib`; cat > "$lib.xml")
> (cd $gitrepo; git commit -m"Package: $package" "$lib.xml")
> done
A way to gather this easily with the current framework would be to allow
BRANCH to refer to a local clone of the abirepo and then let
fedabipkgdiff stick its temporary files in there so that it could easily
be committed. That way when I do one of my tests which does a self
comparison for every RPM, it generates the full data set.
> and rerun that occasionally as updates flow down from the distro.
> This could be done on a single beefy box running containers with
> different distros.
>
> Another flavour could be to take a set of RPM/etc. archives on a
> filesystem (or an ISO image), incrementally decompress them, run abidw
> on the individual files, and similarly construct the git repo of
> abixml files. (This is kind of like how debuginfod produces indexes
> from a bunch of RPMs.)
>
> No matter how the local git repo is populated, each branch describing
> a data contributor's distro could be pushed to the central one,
> bringing that one up to date. Patches representing updates could be
> emailed too, but no one will want to read/review that stuff. We'd
> probably need a trusted pool of contributors who can just commit to
> areas of the central git repo. Secured with gitsigur of course. :-)
>
> The central repo could be built up entirely gradually. If some
> libraries were omitted from initial commits for a distro, a later
> contribution could fill in the gaps.
>
> --------------------
>
> OK, how reasonable does all this sound?
>
>
> - FChE
>
^ permalink raw reply [flat|nested] 9+ messages in thread