The end of 2,1's development is neigh. With all the work that Dodji has 
been doing I feared regressions and so I did a full distro test. So 
where are we:

The number of wrongly compared seems to be really good: None of these 
are familiar to me and so I think we basically solved most of my 
previous issues but introduced some new ones. The vast majority of these 
seem to be C++.

 1. bpftrace - 95 function 1 variable filtered
 2. crash - only unspecified changes on s390
 3. freefem++ - C++ problem one changed virtual function.
 4. gcc - 1 function change filtered out only affects armv7hl
 5. gdb-headless - assert and insanely deep changes on i686 but no
    output for several of the arches. This one probably needs to be
    looked at.
 6. harfbuzz - one function change affecting several arches. Looks easy.
 7. kdepim-addons - C++ type name changes in vtable
 8. kf5-libktorrent - C++ type instability which seems to affect some
    arches. One thing that makes this kind of concerning is for the
    unaffected arches there is no output. This suggests a crash that
    didn't assert.
 9. kf5-mailcommon - similar to kf5-libktorrent
10. ldc - similar to  freefem++
11. mozjs68 - C++ looks like a deep C++ instability issue
12. mozjs91
13. nest - 4 filtered variables similar to bpftrace
14. pocl - 19 filtered out similar to bpftrace
15. vsomeip3 - C++ type confusion
16. lammps - C++probably the best example of type confusion and one of
    our codes
17. smesh - simple C++ type confusion
18. elightenment - 267 functions filtered out
19. gnumeric - fails on armv7hl but the other arches look suspicious
    like a crashes
20. kscope - c++ type instability
21. libcouchbase - 2 changed functions and one filtered
22. tdlib - C++
23. vsomeip3 - more c++ screwyness

Asserts abd segv's This is a bit of a problem. I think that these should 
be beat down before release.

  * abipkgdiff: ../../../libabigail/src/abg-default-reporter.cc:375:
    virtual void abigail::comparison::default_reporter::report(const
    abigail::comparison::qualified_type_diff&, std::ostream&, const
    std::string&) const: Assertion `__abg_cond__' failed.
      o gdb-headless
      o avr-gcc
      o qt6-qtdeclarative
  * abipkgdiff: ../../../libabigail/src/abg-dwarf-reader.cc:17338: bool
    abigail::dwarf_reader::get_soname_of_elf_file(const std::string&,
    std::string&): Assertion `__abg_cond__' failed.
      o irsim
      o julia
  * abipkgdiff: ../../../libabigail/src/abg-ir.cc:17328: uint64_t
    abigail::ir::array_type_def::subrange_type::get_length() const:
    Assertion `__abg_cond__' failed.
      o GtkAda3
      o aws
      o ghdl
      o GtkAda3
      o GtkAda
      o templates_parser
      o xmlada
  * abipkgdiff: ../../../libabigail/src/abg-ir.cc:26131:
    abigail::ir::type_base* abigail::ir::get_exemplar_type(const
    type_base*): Assertion `__abg_cond__' failed.
      o avr-binutils
      o bpftrace
      o firebird
      o llvm8.0
      o llvm
      o mozjs68
      o mozjs78
      o mozjs91
      o xorg-x11-server-Xorg
  * abipkgdiff: ../../../libabigail/src/abg-ir.cc:26191: size_t
    abigail::ir::hash_as_canonical_type_or_constant(const type_base*):
    Assertion `__abg_cond__' failed.
      o kcov
      o gnumeric
  * abipkgdiff: ../../../libabigail/src/abg-reader.cc:3400:
    abigail::ir::string_elf_symbols_map_sptr
    abigail::xml_reader::build_elf_symbol_db(read_context&, xmlNodePtr,
    bool): Assertion `__abg_cond__' failed.

      o alt-ergo
      o bibtex2html
      o coccinelle
      o frama-c
      o freetennis
      o guestfs-tools
      o haxe
      o hevea
      o ocaml-atd
      o ocaml-biniou
      o ocaml-bisect-ppx
      o ocaml-camlp5
      o ocaml-cinaps
      o ocaml-cppo
      o ocaml-csv
      o ocaml-cudf
      o ocaml-dune
      o ocaml-findlib
      o ocaml-gettext
      o ocamlify
      o ocaml-lablgtk3
      o ocaml-lambda-term
      o ocaml-mdx
      o ocaml-menhir
      o ocaml-merlin
      o ocamlmod
      o ocaml-obuild
      o ocaml-ocamlbuild
      o ocaml-ocamlnet
      o ocaml-ocp-indent
      o ocaml-octavius
      o ocaml-odoc
      o ocaml-omake
      o ocaml
      o ocaml-ppx-assert
      o ocaml-ppx-base
      o ocaml-ppx-cold
      o ocaml-ppx-custom-printf
      o ocaml-ppx-deriving
      o ocaml-ppx-expect
      o ocaml-ppx-here
      o ocaml-ppx-inline-test
      o ocaml-ppx-js-style
      o ocaml-ppx-let
      o ocaml-ppxlib
      o ocaml-psmt2-frontend
      o ocaml-qtest
      o ocaml-sedlex
      o ocaml-topkg
      o ocaml-xmlm
      o ocaml-yojson
      o opam
      o planets
      o prooftree
      o supermin
      o virt-top
      o virt-v2v
      o why3
      o zenon

Segfault crashes. They fall into 4 groups. Two of which only happen 
once. The worst one is the add_data_member problem:

 1. apptainer- likely infinite recursion
     1. rclone
     2. apptainer
     3. singularity
     4. golang-k8s-sample-cli-plugin
     5. stargz-snapshotter- only on x86_64
 2. abigail::ir::class_or_union::add_data_member ()
     1. mediawriter
     2. qt6-qtsensors
     3. qt6-qtscxml
     4. qgnomeplatform-qt6
     5. AusweisApp2
     6. qt6-qtremoteobjects
     7. qt6-qtvirtualkeyboard
     8. qt6-qtserialbus
     9. qt6-qtwayland
    10. qt6-qtconnectivity
    11. qt6-qt5compat
 3. containerd- only on x86_64
 4. bettercap - unique


      mediawriter

Thread 6 "abipkgdiff" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff61fd640 (LWP 1060032)]
abigail::ir::class_or_union::add_data_member (this=0x0, 
v=std::shared_ptr<abigail::ir::var_decl> (use count 2, weak count 0) = 
{...}, access=abigail::ir::public_access, is_laid_out=<optimized out>, 
is_static=<optimized out>, offset_in_bits=0) at 
../../../libabigail/src/abg-ir.cc:21715
21715      priv_->data_members_.push_back(v);
#0  abigail::ir::class_or_union::add_data_member (this=0x0, 
v=std::shared_ptr<abigail::ir::var_decl> (use count 2, weak count 0) = 
{...}, access=abigail::ir::public_access, is_laid_out=<optimized out>, 
is_static=<optimized out>, offset_in_bits=0) at 
../../../libabigail/src/abg-ir.cc:21715
#1  0x00007ffff7d00387 in abigail::ir::var_decl::clone (this=<optimized 
out>) at ../../../libabigail/src/abg-ir.cc:19135
#2  0x00007ffff7e37491 in abigail::dwarf_reader::build_ir_node_from_die 
(ctxt=..., die=0x7ffff61fc1e0, scope=<optimized out>, 
called_from_public_decl=<optimized out>, where_offset=88550, 
is_declaration_only=false, is_required_decl_spec=false) at 
/usr/include/c++/12/bits/shared_ptr_base.h:1665
#3  0x00007ffff7e38b34 in abigail::dwarf_reader::build_ir_node_from_die 
(ctxt=..., die=0x7ffff61fc1e0, called_from_public_decl=<optimized out>, 
where_offset=88550) at /usr/include/c++/12/bits/shared_ptr_base.h:1665
#4  0x00007ffff7e37869 in 
abigail::dwarf_reader::build_namespace_decl_and_add_to_ir 
(where_offset=88550, die=0x7ffff61fc3f0, ctxt=...) at 
../../../libabigail/src/abg-dwarf-reader.cc:12901
#5  abigail::dwarf_reader::build_ir_node_from_die (ctxt=..., 
die=0x7ffff61fc3f0, scope=<optimized out>, 
called_from_public_decl=<optimized out>, where_offset=88550, 
is_declaration_only=<optimized out>, is_required_decl_spec=false) at 
../../../libabigail/src/abg-dwarf-reader.cc:16481
#6  0x00007ffff7e38b34 in abigail::dwarf_reader::build_ir_node_from_die 
(ctxt=..., die=0x7ffff61fc3f0, called_from_public_decl=<optimized out>, 
where_offset=88550) at /usr/include/c++/12/bits/shared_ptr_base.h:1665
#7  0x00007ffff7e2769a in 
abigail::dwarf_reader::build_translation_unit_and_add_to_ir (ctxt=..., 
die=<optimized out>, address_size=<optimized out>) at 
../../../libabigail/src/abg-dwarf-reader.cc:12784
#8  0x00007ffff7e4192e in 
abigail::dwarf_reader::read_debug_info_into_corpus (ctxt=...) at 
../../../libabigail/src/abg-dwarf-reader.cc:15881
#9  abigail::dwarf_reader::read_corpus_from_elf (ctxt=..., 
status=@0x7ffff61fc9d4: abigail::elf_reader::STATUS_UNKNOWN) at 
../../../libabigail/src/abg-dwarf-reader.cc:17016
#10 0x00000000004100dc in compare_to_self 
(detailed_error_status=<synthetic pointer>, ctxt=<error reading 
variable: access outside bounds of object referenced via synthetic 
pointer>, diff=<error reading variable: access outside bounds of object 
referenced via synthetic pointer>, env=<error reading variable: access 
outside bounds of object referenced via synthetic pointer>, opts=..., 
debug_dir=..., elf=...) at ../../../libabigail/tools/abipkgdiff.cc:1625
#11 self_compare_task::perform (this=0x43b720) at 
../../../libabigail/tools/abipkgdiff.cc:2163
#12 0x00007ffff7e5fb2d in 
abigail::workers::worker::wait_to_execute_a_task (p=0x43af30) at 
../../../libabigail/src/abg-workers.cc:400
#13 0x00007ffff728ce2d in start_thread (arg=<optimized out>) at 
pthread_create.c:442
#14 0x00007ffff73121b0 in clone3 () at 
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81


      bettercap

[ben@alien decks]$ gdb `which abipkgdiff`
(gdb) set args --self-check  --d1 
/home/ben/.cache/fedabipkgdiff/downloads/bettercap-debuginfo-2.32.0-5.fc36.aarch64.rpm 
/home/ben/.cache/fedabipkgdiff/downloads/bettercap-2.32.0-5.fc36.aarch64.rpm
(gdb) r
Starting program: 
/home/ben/Shared/Work/test/libabigail-x86_64/bin/abipkgdiff 
--self-check  --d1 
/home/ben/.cache/fedabipkgdiff/downloads/bettercap-debuginfo-2.32.0-5.fc36.aarch64.rpm 
/home/ben/.cache/fedabipkgdiff/downloads/bettercap-2.32.0-5.fc36.aarch64.rpm
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff71ff640 (LWP 1048200)]
[New Thread 0x7ffff69fe640 (LWP 1048201)]
[Detaching after vfork from child process 1048202]
[Detaching after vfork from child process 1048203]
[Thread 0x7ffff69fe640 (LWP 1048201) exited]
[Thread 0x7ffff71ff640 (LWP 1048200) exited]
[Detaching after vfork from child process 1048210]
[New Thread 0x7ffff69fe640 (LWP 1048213)]

Thread 4 "abipkgdiff" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff69fe640 (LWP 1048213)]
0x00007ffff7a0e0cb in Dwarf_Abbrev_Hash_find 
(htab=htab@entry=0x7fffea502a10, hval=hval@entry=58) at 
../lib/dynamicsizehash_concurrent.c:461
Downloading 0.00 MB source file 
/usr/src/debug/elfutils-0.187-4.fc36.x86_64/libdw/../lib/dynamicsizehash_concurrent.c
461      while (pthread_rwlock_tryrdlock(&htab->resize_rwl) != 0)


      containerd

Thread 5 "abipkgdiff" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff71ff640 (LWP 1077737)]
0x00007ffff7d0545c in 
abigail::ir::function_decl::get_pretty_representation_of_declarator[abi:cxx11](bool) 
const (this=0x7fffcee02bc0, internal=true) at 
../../../libabigail/src/abg-ir.cc:20487
20487          result += type_decl->get_qualified_name(internal);
#0  0x00007ffff7d0545c in 
abigail::ir::function_decl::get_pretty_representation_of_declarator[abi:cxx11](bool) 
const (this=0x7fffcee02bc0, internal=true) at 
../../../libabigail/src/abg-ir.cc:20487
#1  0x00007ffff7d05d1a in 
abigail::ir::function_decl::get_pretty_representation[abi:cxx11](bool, 
bool) const (this=0x7fffcee02bc0, internal=<optimized out>) at 
../../../libabigail/src/abg-ir.cc:20423
#2  0x00007ffff7cdb58f in 
abigail::ir::get_pretty_representation[abi:cxx11](abigail::ir::decl_base 
const*, bool) (d=<optimized out>, internal=<optimized out>) at 
../../../libabigail/src/abg-ir.cc:9175
#3  0x00007ffff7cdb5f7 in 
abigail::ir::get_pretty_representation[abi:cxx11](std::shared_ptr<abigail::ir::decl_base> 
const&, bool) (d=..., internal=<optimized out>) at 
/usr/include/c++/12/bits/shared_ptr_base.h:1665
#4  0x00007ffff7e4b45f in 
abigail::xml_writer::write_context::decl_is_emitted 
(decl=std::shared_ptr<abigail::ir::decl_base> (use count 3, weak count 
0) = {...}, this=0x7fffcdf3d200) at 
../../../libabigail/src/abg-writer.cc:810
#5  abigail::xml_writer::write_translation_unit (ctxt=..., tu=..., 
indent=2, is_last=<optimized out>) at 
../../../libabigail/src/abg-writer.cc:2467
#6  0x00007ffff7e52338 in abigail::xml_writer::write_corpus (ctxt=..., 
corpus=std::shared_ptr<abigail::ir::corpus> (use count 2, weak count 0) 
= {...}, indent=indent@entry=0, 
member_of_group=member_of_group@entry=false) at 
../../../libabigail/src/abg-writer.cc:4585
#7  0x00000000004115a7 in compare_to_self 
(detailed_error_status=<synthetic pointer>, ctxt=<error reading 
variable: access outside bounds of object referenced via synthetic 
pointer>, diff=<error reading variable: access outside bounds of object 
referenced via synthetic pointer>, env=<error reading variable: access 
outside bounds of object referenced via synthetic pointer>, opts=..., 
debug_dir=..., elf=...) at ../../../libabigail/tools/abipkgdiff.cc:1677
#8  self_compare_task::perform (this=0x43f320) at 
../../../libabigail/tools/abipkgdiff.cc:2163
#9  0x00007ffff7e5fb2d in 
abigail::workers::worker::wait_to_execute_a_task (p=0x43af30) at 
../../../libabigail/src/abg-workers.cc:400
#10 0x00007ffff728ce2d in start_thread (arg=<optimized out>) at 
pthread_create.c:442
#11 0x00007ffff73121b0 in clone3 () at 
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81