From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 1F6333858D35 for ; Wed, 15 Nov 2023 15:53:10 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1F6333858D35 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1F6333858D35 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700063591; cv=none; b=im+vBE7mqaoOoAeEjG5ONRt6JCsrYKw7UTTjkqIv1j08EPCtLJvDyYuRiWKF/jIIwoCgqnH75Gsr8O9J1CVKUZJ587Dy88LOttfkw/BC4JRTLs+79OqmNm9si2OSzf6RAJ0SnxEwqYk5tqiXXpT0NYCJglVvqvJVAwJK+VudhEY= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1700063591; c=relaxed/simple; bh=/F8bwZefEC59SLMIoDB9yxgUsIHdlfuWIYk4c18eQQQ=; h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version; b=BUMm9kM7DGtPZs2Tn2x+UBAckGz/d32gPEbwor6pw6Xh8SWbAwgkDp0HhuHDK1l8bIB9AydGlnJOfPoH8Z1Ibwi1eaqIWQev75mvGXWszRQK201UM52P61zrZmn+DjDMRbPhzCDpIY2tnAQmcQir4Amr+aIM19/zFm4SktMiopQ= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700063589; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=cYZ0pf8QSzGHNhmp91GBqDSI/DVVP97X083qJ4rvrFM=; b=HNifk+bWRawFfNXv+uLlr37qDxzGXwL+jlubbbjtT4ZAuukBH94rci6VyBBj4vHXIQPq6h 1fsFmGMyzjdF2vTiVBFV/svifFw9EJ70Dp+ceKNtnFdzZYY4VP1JW3vcIzWJfr4Z/94I2r MtWyY75IiabF6NKb6pPR5zGlf14a1hY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-134-rLxp42vwOVa0yOrNjZ5LXA-1; Wed, 15 Nov 2023 10:53:08 -0500 X-MC-Unique: rLxp42vwOVa0yOrNjZ5LXA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 057E0811E7D for ; Wed, 15 Nov 2023 15:53:08 +0000 (UTC) Received: from redhat.com (unknown [10.22.8.8]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DBE1D36EE for ; Wed, 15 Nov 2023 15:53:07 +0000 (UTC) Received: from fche by redhat.com with local (Exim 4.94.2) (envelope-from ) id 1r3ICQ-0005ks-W2 for libabigail@sourceware.org; Wed, 15 Nov 2023 10:53:07 -0500 Date: Wed, 15 Nov 2023 10:53:06 -0500 From: "Frank Ch. Eigler" To: libabigail@sourceware.org Subject: idea: abigail abixml archive Message-ID: <20231115155306.GC15862@redhat.com> MIME-Version: 1.0 User-Agent: Mutt/1.12.0 (2019-05-25) X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,KAM_ASCII_DIVIDERS,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi - I'd love some feedback about the following idea, related to using libabigail to assemble a crowdsourced database of abixml files for linux distros. The germ of the idea is that developers may need to know whether a binary they built or found is likely to be abi-compatible with a given distro / version. This is possible today by downloading the target distro binaries and running libabigail locally against them, or using front-end scripts like fedabipkgdiff that do the downloading first. But this is a pain if one wants to compare against a range of versions or foreign distros. So the idea is instead to let people use an public archive of abixml artifacts instead of the binaries. The abixml files are relatively tiny, barely-ever changing, and should be an effective proxy for the real binaries. It's just a small matter of (a) storing, (b) using, and (c) collecting it. -------------------- For storing this data, I envision overloading the libabigil git repo (or a new one) with storage of the abixml documents. To keep it dead simple, there could be one branch per /etc/os-release $ID/$VERSION_ID, one file per shared library in the distribution. For example, a fedora-39-x86-64 copy of /usr/lib64/libc.so.6, the file abidw produces could sit at repo git://sourceware.org/git/libabigail.git branch gitabixml/fedora/39/x86_64 file /usr/lib64/libc.so.6.xml (Symlinks in the distro fs could be represented as symlinks in git.) Updates to the distro package of course happen. It seems natural to update the abixml file for the affected file(s) right there in place. Since it may sometimes be desirable to track what package version (e.g. rpm n-v-r) is associated with the abixml data of a given version, we could use stylized records in the git commit text (or a git note, or maybe a tag). That would mean one git commit per updated package, with metadata message like: Package: glibc-2.38-7.fc39.x86_64 Maybe abidw version tags would be useful to add. -------------------- For using this data, I envision abidiff / abicompat taking a new form for its right operand. It could be a git url identifying the distro branch or tag. libabigail would fetch the corresponding file.xml within that. Simplify/default the heck out of it for ease of use: export $BRANCH=fedora/39/x86_64 abicompat /bin/myprogram gitabixml:$BRANCH (Where "gitabixml:" could instruct the tool to look at the sourceware libabigail git / gitweb / cgit server. Let users specify different or private git servers via environment variables or something. -------------------- For collecting this data, I envision writing some distro-specific scripts, kind of like fedabipkgdiff, being run by contributors or ourselves. One flavour could run in operational installed distros, doing the equivalent of find $PATHS -name '*.so.*' | while read lib; do # or filter with elfclassify package=`rpm -qf "$lib"` abidw "$lib" | (cd $gitrepo/`dirname $lib`; cat > "$lib.xml") (cd $gitrepo; git commit -m"Package: $package" "$lib.xml") done and rerun that occasionally as updates flow down from the distro. This could be done on a single beefy box running containers with different distros. Another flavour could be to take a set of RPM/etc. archives on a filesystem (or an ISO image), incrementally decompress them, run abidw on the individual files, and similarly construct the git repo of abixml files. (This is kind of like how debuginfod produces indexes from a bunch of RPMs.) No matter how the local git repo is populated, each branch describing a data contributor's distro could be pushed to the central one, bringing that one up to date. Patches representing updates could be emailed too, but no one will want to read/review that stuff. We'd probably need a trusted pool of contributors who can just commit to areas of the central git repo. Secured with gitsigur of course. :-) The central repo could be built up entirely gradually. If some libraries were omitted from initial commits for a distro, a later contribution could fill in the gaps. -------------------- OK, how reasonable does all this sound? - FChE