Progress! With Dodji's latest patches I took the fails from the previous 
test and reran with the latest code. 80 packages which previously failed 
now pass.

The class of "wrongly compared" seems to have continued to improve. 
There seem to be two classes of problems. One is just unspecified 
filtered symbols. The other class of problems are related to some kind 
of type confusion in C++.

 1. crash
 2. enlightenmentd
 3. freefem++
 4. gimp
 5. harfbuzz
 6. kdepim-addons
 7. kf5-libktorrent
 8. kf5-mailcommon
 9. kscope
10. ldc
11. libcouchbase
12. nest
13. pocl
14. qt6-qtbase
15. smesh
16. tdlib
17. vsomeip3

The asserts appear to be about the same but some of the line numbers 
have changed slightly but that is it. This affects 72 packages.

The segv's are all gone yea! One interesting thing though is a few of 
the programs that were segv'ing are now returning 68. I have yet to 
figure out what exit code 68 means:

AusweisApp2 - Unknown 68
kcov - Unknown 68
bettercap - Unknown 68
singularity - Unknown 68
stargz-snapshotter - Unknown 68
llvm11 - Unknown 68
qt6-qtwayland - Unknown 68
flang - Unknown 68

-ben

On 9/21/22 18:19, Ben Woodard wrote:
>
> The end of 2,1's development is neigh. With all the work that Dodji 
> has been doing I feared regressions and so I did a full distro test. 
> So where are we:
>
> The number of wrongly compared seems to be really good: None of these 
> are familiar to me and so I think we basically solved most of my 
> previous issues but introduced some new ones. The vast majority of 
> these seem to be C++.
>
>  1. bpftrace - 95 function 1 variable filtered
>  2. crash - only unspecified changes on s390
>  3. freefem++ - C++ problem one changed virtual function.
>  4. gcc - 1 function change filtered out only affects armv7hl
>  5. gdb-headless - assert and insanely deep changes on i686 but no
>     output for several of the arches. This one probably needs to be
>     looked at.
>  6. harfbuzz - one function change affecting several arches. Looks easy.
>  7. kdepim-addons - C++ type name changes in vtable
>  8. kf5-libktorrent - C++ type instability which seems to affect some
>     arches. One thing that makes this kind of concerning is for the
>     unaffected arches there is no output. This suggests a crash that
>     didn't assert.
>  9. kf5-mailcommon - similar to kf5-libktorrent
> 10. ldc - similar to  freefem++
> 11. mozjs68 - C++ looks like a deep C++ instability issue
> 12. mozjs91
> 13. nest - 4 filtered variables similar to bpftrace
> 14. pocl - 19 filtered out similar to bpftrace
> 15. vsomeip3 - C++ type confusion
> 16. lammps - C++probably the best example of type confusion and one of
>     our codes
> 17. smesh - simple C++ type confusion
> 18. elightenment - 267 functions filtered out
> 19. gnumeric - fails on armv7hl but the other arches look suspicious
>     like a crashes
> 20. kscope - c++ type instability
> 21. libcouchbase - 2 changed functions and one filtered
> 22. tdlib - C++
> 23. vsomeip3 - more c++ screwyness
>
> Asserts abd segv's This is a bit of a problem. I think that these 
> should be beat down before release.
>
>   * abipkgdiff: ../../../libabigail/src/abg-default-reporter.cc:375:
>     virtual void abigail::comparison::default_reporter::report(const
>     abigail::comparison::qualified_type_diff&, std::ostream&, const
>     std::string&) const: Assertion `__abg_cond__' failed.
>       o gdb-headless
>       o avr-gcc
>       o qt6-qtdeclarative
>   * abipkgdiff: ../../../libabigail/src/abg-dwarf-reader.cc:17338:
>     bool abigail::dwarf_reader::get_soname_of_elf_file(const
>     std::string&, std::string&): Assertion `__abg_cond__' failed.
>       o irsim
>       o julia
>   * abipkgdiff: ../../../libabigail/src/abg-ir.cc:17328: uint64_t
>     abigail::ir::array_type_def::subrange_type::get_length() const:
>     Assertion `__abg_cond__' failed.
>       o GtkAda3
>       o aws
>       o ghdl
>       o GtkAda3
>       o GtkAda
>       o templates_parser
>       o xmlada
>   * abipkgdiff: ../../../libabigail/src/abg-ir.cc:26131:
>     abigail::ir::type_base* abigail::ir::get_exemplar_type(const
>     type_base*): Assertion `__abg_cond__' failed.
>       o avr-binutils
>       o bpftrace
>       o firebird
>       o llvm8.0
>       o llvm
>       o mozjs68
>       o mozjs78
>       o mozjs91
>       o xorg-x11-server-Xorg
>   * abipkgdiff: ../../../libabigail/src/abg-ir.cc:26191: size_t
>     abigail::ir::hash_as_canonical_type_or_constant(const type_base*):
>     Assertion `__abg_cond__' failed.
>       o kcov
>       o gnumeric
>   * abipkgdiff: ../../../libabigail/src/abg-reader.cc:3400:
>     abigail::ir::string_elf_symbols_map_sptr
>     abigail::xml_reader::build_elf_symbol_db(read_context&,
>     xmlNodePtr, bool): Assertion `__abg_cond__' failed.
>
>       o alt-ergo
>       o bibtex2html
>       o coccinelle
>       o frama-c
>       o freetennis
>       o guestfs-tools
>       o haxe
>       o hevea
>       o ocaml-atd
>       o ocaml-biniou
>       o ocaml-bisect-ppx
>       o ocaml-camlp5
>       o ocaml-cinaps
>       o ocaml-cppo
>       o ocaml-csv
>       o ocaml-cudf
>       o ocaml-dune
>       o ocaml-findlib
>       o ocaml-gettext
>       o ocamlify
>       o ocaml-lablgtk3
>       o ocaml-lambda-term
>       o ocaml-mdx
>       o ocaml-menhir
>       o ocaml-merlin
>       o ocamlmod
>       o ocaml-obuild
>       o ocaml-ocamlbuild
>       o ocaml-ocamlnet
>       o ocaml-ocp-indent
>       o ocaml-octavius
>       o ocaml-odoc
>       o ocaml-omake
>       o ocaml
>       o ocaml-ppx-assert
>       o ocaml-ppx-base
>       o ocaml-ppx-cold
>       o ocaml-ppx-custom-printf
>       o ocaml-ppx-deriving
>       o ocaml-ppx-expect
>       o ocaml-ppx-here
>       o ocaml-ppx-inline-test
>       o ocaml-ppx-js-style
>       o ocaml-ppx-let
>       o ocaml-ppxlib
>       o ocaml-psmt2-frontend
>       o ocaml-qtest
>       o ocaml-sedlex
>       o ocaml-topkg
>       o ocaml-xmlm
>       o ocaml-yojson
>       o opam
>       o planets
>       o prooftree
>       o supermin
>       o virt-top
>       o virt-v2v
>       o why3
>       o zenon
>
> Segfault crashes. They fall into 4 groups. Two of which only happen 
> once. The worst one is the add_data_member problem:
>
>  1. apptainer- likely infinite recursion
>      1. rclone
>      2. apptainer
>      3. singularity
>      4. golang-k8s-sample-cli-plugin
>      5. stargz-snapshotter- only on x86_64
>  2. abigail::ir::class_or_union::add_data_member ()
>      1. mediawriter
>      2. qt6-qtsensors
>      3. qt6-qtscxml
>      4. qgnomeplatform-qt6
>      5. AusweisApp2
>      6. qt6-qtremoteobjects
>      7. qt6-qtvirtualkeyboard
>      8. qt6-qtserialbus
>      9. qt6-qtwayland
>     10. qt6-qtconnectivity
>     11. qt6-qt5compat
>  3. containerd- only on x86_64
>  4. bettercap - unique
>
>
>       mediawriter
>
> Thread 6 "abipkgdiff" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff61fd640 (LWP 1060032)]
> abigail::ir::class_or_union::add_data_member (this=0x0, 
> v=std::shared_ptr<abigail::ir::var_decl> (use count 2, weak count 0) = 
> {...}, access=abigail::ir::public_access, is_laid_out=<optimized out>, 
> is_static=<optimized out>, offset_in_bits=0) at 
> ../../../libabigail/src/abg-ir.cc:21715
> 21715      priv_->data_members_.push_back(v);
> #0  abigail::ir::class_or_union::add_data_member (this=0x0, 
> v=std::shared_ptr<abigail::ir::var_decl> (use count 2, weak count 0) = 
> {...}, access=abigail::ir::public_access, is_laid_out=<optimized out>, 
> is_static=<optimized out>, offset_in_bits=0) at 
> ../../../libabigail/src/abg-ir.cc:21715
> #1  0x00007ffff7d00387 in abigail::ir::var_decl::clone 
> (this=<optimized out>) at ../../../libabigail/src/abg-ir.cc:19135
> #2  0x00007ffff7e37491 in 
> abigail::dwarf_reader::build_ir_node_from_die (ctxt=..., 
> die=0x7ffff61fc1e0, scope=<optimized out>, 
> called_from_public_decl=<optimized out>, where_offset=88550, 
> is_declaration_only=false, is_required_decl_spec=false) at 
> /usr/include/c++/12/bits/shared_ptr_base.h:1665
> #3  0x00007ffff7e38b34 in 
> abigail::dwarf_reader::build_ir_node_from_die (ctxt=..., 
> die=0x7ffff61fc1e0, called_from_public_decl=<optimized out>, 
> where_offset=88550) at /usr/include/c++/12/bits/shared_ptr_base.h:1665
> #4  0x00007ffff7e37869 in 
> abigail::dwarf_reader::build_namespace_decl_and_add_to_ir 
> (where_offset=88550, die=0x7ffff61fc3f0, ctxt=...) at 
> ../../../libabigail/src/abg-dwarf-reader.cc:12901
> #5  abigail::dwarf_reader::build_ir_node_from_die (ctxt=..., 
> die=0x7ffff61fc3f0, scope=<optimized out>, 
> called_from_public_decl=<optimized out>, where_offset=88550, 
> is_declaration_only=<optimized out>, is_required_decl_spec=false) at 
> ../../../libabigail/src/abg-dwarf-reader.cc:16481
> #6  0x00007ffff7e38b34 in 
> abigail::dwarf_reader::build_ir_node_from_die (ctxt=..., 
> die=0x7ffff61fc3f0, called_from_public_decl=<optimized out>, 
> where_offset=88550) at /usr/include/c++/12/bits/shared_ptr_base.h:1665
> #7  0x00007ffff7e2769a in 
> abigail::dwarf_reader::build_translation_unit_and_add_to_ir (ctxt=..., 
> die=<optimized out>, address_size=<optimized out>) at 
> ../../../libabigail/src/abg-dwarf-reader.cc:12784
> #8  0x00007ffff7e4192e in 
> abigail::dwarf_reader::read_debug_info_into_corpus (ctxt=...) at 
> ../../../libabigail/src/abg-dwarf-reader.cc:15881
> #9  abigail::dwarf_reader::read_corpus_from_elf (ctxt=..., 
> status=@0x7ffff61fc9d4: abigail::elf_reader::STATUS_UNKNOWN) at 
> ../../../libabigail/src/abg-dwarf-reader.cc:17016
> #10 0x00000000004100dc in compare_to_self 
> (detailed_error_status=<synthetic pointer>, ctxt=<error reading 
> variable: access outside bounds of object referenced via synthetic 
> pointer>, diff=<error reading variable: access outside bounds of 
> object referenced via synthetic pointer>, env=<error reading variable: 
> access outside bounds of object referenced via synthetic pointer>, 
> opts=..., debug_dir=..., elf=...) at 
> ../../../libabigail/tools/abipkgdiff.cc:1625
> #11 self_compare_task::perform (this=0x43b720) at 
> ../../../libabigail/tools/abipkgdiff.cc:2163
> #12 0x00007ffff7e5fb2d in 
> abigail::workers::worker::wait_to_execute_a_task (p=0x43af30) at 
> ../../../libabigail/src/abg-workers.cc:400
> #13 0x00007ffff728ce2d in start_thread (arg=<optimized out>) at 
> pthread_create.c:442
> #14 0x00007ffff73121b0 in clone3 () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
>
>
>       bettercap
>
> [ben@alien decks]$ gdb `which abipkgdiff`
> (gdb) set args --self-check  --d1 
> /home/ben/.cache/fedabipkgdiff/downloads/bettercap-debuginfo-2.32.0-5.fc36.aarch64.rpm 
> /home/ben/.cache/fedabipkgdiff/downloads/bettercap-2.32.0-5.fc36.aarch64.rpm
> (gdb) r
> Starting program: 
> /home/ben/Shared/Work/test/libabigail-x86_64/bin/abipkgdiff 
> --self-check  --d1 
> /home/ben/.cache/fedabipkgdiff/downloads/bettercap-debuginfo-2.32.0-5.fc36.aarch64.rpm 
> /home/ben/.cache/fedabipkgdiff/downloads/bettercap-2.32.0-5.fc36.aarch64.rpm
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> [New Thread 0x7ffff71ff640 (LWP 1048200)]
> [New Thread 0x7ffff69fe640 (LWP 1048201)]
> [Detaching after vfork from child process 1048202]
> [Detaching after vfork from child process 1048203]
> [Thread 0x7ffff69fe640 (LWP 1048201) exited]
> [Thread 0x7ffff71ff640 (LWP 1048200) exited]
> [Detaching after vfork from child process 1048210]
> [New Thread 0x7ffff69fe640 (LWP 1048213)]
>
> Thread 4 "abipkgdiff" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff69fe640 (LWP 1048213)]
> 0x00007ffff7a0e0cb in Dwarf_Abbrev_Hash_find 
> (htab=htab@entry=0x7fffea502a10, hval=hval@entry=58) at 
> ../lib/dynamicsizehash_concurrent.c:461
> Downloading 0.00 MB source file 
> /usr/src/debug/elfutils-0.187-4.fc36.x86_64/libdw/../lib/dynamicsizehash_concurrent.c
> 461      while (pthread_rwlock_tryrdlock(&htab->resize_rwl) != 0)
>
>
>       containerd
>
> Thread 5 "abipkgdiff" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff71ff640 (LWP 1077737)]
> 0x00007ffff7d0545c in 
> abigail::ir::function_decl::get_pretty_representation_of_declarator[abi:cxx11](bool) 
> const (this=0x7fffcee02bc0, internal=true) at 
> ../../../libabigail/src/abg-ir.cc:20487
> 20487          result += type_decl->get_qualified_name(internal);
> #0  0x00007ffff7d0545c in 
> abigail::ir::function_decl::get_pretty_representation_of_declarator[abi:cxx11](bool) 
> const (this=0x7fffcee02bc0, internal=true) at 
> ../../../libabigail/src/abg-ir.cc:20487
> #1  0x00007ffff7d05d1a in 
> abigail::ir::function_decl::get_pretty_representation[abi:cxx11](bool, 
> bool) const (this=0x7fffcee02bc0, internal=<optimized out>) at 
> ../../../libabigail/src/abg-ir.cc:20423
> #2  0x00007ffff7cdb58f in 
> abigail::ir::get_pretty_representation[abi:cxx11](abigail::ir::decl_base 
> const*, bool) (d=<optimized out>, internal=<optimized out>) at 
> ../../../libabigail/src/abg-ir.cc:9175
> #3  0x00007ffff7cdb5f7 in 
> abigail::ir::get_pretty_representation[abi:cxx11](std::shared_ptr<abigail::ir::decl_base> 
> const&, bool) (d=..., internal=<optimized out>) at 
> /usr/include/c++/12/bits/shared_ptr_base.h:1665
> #4  0x00007ffff7e4b45f in 
> abigail::xml_writer::write_context::decl_is_emitted 
> (decl=std::shared_ptr<abigail::ir::decl_base> (use count 3, weak count 
> 0) = {...}, this=0x7fffcdf3d200) at 
> ../../../libabigail/src/abg-writer.cc:810
> #5  abigail::xml_writer::write_translation_unit (ctxt=..., tu=..., 
> indent=2, is_last=<optimized out>) at 
> ../../../libabigail/src/abg-writer.cc:2467
> #6  0x00007ffff7e52338 in abigail::xml_writer::write_corpus (ctxt=..., 
> corpus=std::shared_ptr<abigail::ir::corpus> (use count 2, weak count 
> 0) = {...}, indent=indent@entry=0, 
> member_of_group=member_of_group@entry=false) at 
> ../../../libabigail/src/abg-writer.cc:4585
> #7  0x00000000004115a7 in compare_to_self 
> (detailed_error_status=<synthetic pointer>, ctxt=<error reading 
> variable: access outside bounds of object referenced via synthetic 
> pointer>, diff=<error reading variable: access outside bounds of 
> object referenced via synthetic pointer>, env=<error reading variable: 
> access outside bounds of object referenced via synthetic pointer>, 
> opts=..., debug_dir=..., elf=...) at 
> ../../../libabigail/tools/abipkgdiff.cc:1677
> #8  self_compare_task::perform (this=0x43f320) at 
> ../../../libabigail/tools/abipkgdiff.cc:2163
> #9  0x00007ffff7e5fb2d in 
> abigail::workers::worker::wait_to_execute_a_task (p=0x43af30) at 
> ../../../libabigail/src/abg-workers.cc:400
> #10 0x00007ffff728ce2d in start_thread (arg=<optimized out>) at 
> pthread_create.c:442
> #11 0x00007ffff73121b0 in clone3 () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81