Hello, In the text below, the term interface means the "declaration of either a function or a global variable". To analyze the compatibility between an application and a dependent library, one has essentially to analyze the interfaces of the application that have undefined symbols (aka undefined interfaces from the application) and see how they relate to the same interfaces coming from the library but with symbols that are defined and exported from the library (aka defined/exported interfaces from the library). An interface that is undefined in the application and defined in the library is an interface that the application consumes from the library. In other words an undefined interface from an application is an interface that the application expects and an interface that is exported by a library is an interface that the library provides. If comparing the expected interface against the provided interface yields a meaningful difference, then that might mean there is an incompatibility between the application and the library on that interface. This patch uses this scheme to re-implement the weak mode of ABI compatibility between an application and a library. The patch adds the concept of undefined functions and variables to the ABI corpus. Then it teaches the DWARF reader how to construct the IR for these undefined interfaces. Then it revisits the weak mode of operation of abicompat to make it compare the interfaces expected from the application against the corresponding interfaces provided by library. The patch also teaches the weak mode abicompat how to detect incompatibilities between the interfaces that plugin expects and the interfaces provided by the application. This patch makes the CTF front-end construct undefined interfaces. A subsequent patch is needed to construct undefined interfaces from the BTF front-end however. * include/abg-corpus.h (corpus::{lookup_variable, get_undefined_functions, get_undefined_variables}): Declare new member functions. (corpus::exported_decls_builder::maybe_add_{fn,var}_to_exported_fns): Make this return a boolean. * include/abg-fe-iface.h (fe_iface::options_type): Add a new load_undefined_interfaces enumerator. (fe_iface::add_fn_to_exported_or_undefined_decls): Rename maybe_add_fn_to_exported_decls into this. (fe_iface::add_var_to_exported_or_undefined_decls): Rename maybe_add_var_to_exported_decls into this. * src/abg-btf-reader.cc (reader::build_ir_node_from_btf_type): Adjust call to maybe_add_fn_to_exported_decls as add_fn_to_exported_or_undefined_decls. Similarly, adjust call to maybe_add_var_to_exported_decls as add_var_to_exported_or_undefined_decls. * src/abg-corpus-priv.h (corpus::priv::undefined_{fns,vars}): Add new member variables. * src/abg-corpus.cc (corpus::exported_decls_builder::maybe_add_{fn,var}_to_exported_fns): Return a bool iff the decl was added to the set of exported decls. (corpus::{lookup_variable, get_undefined_functions, get_undefined_variables}): Define new member functions. (corpus::sort_{functions,variables}): Sort the undefined decls too. * (corpus::lookup_{function,variable}_symbol): Lookup the symbol also among undefined symbols, not just among defined symbols. * src/abg-ctf-reader.cc (reader::process_ctf_archive): Adjust call to maybe_add_fn_to_exported_decls as add_fn_to_exported_or_undefined_decls. Similarly, adjust call to maybe_add_var_to_exported_decls as add_var_to_exported_or_undefined_decls. Also, sort functions & variables in the corpus. * src/abg-dwarf-reader.cc (die_name_and_linkage_name): Define new static function. (reader::fixup_functions_with_no_symbols): Adjust call to maybe_add_fn_to_exported_decls as add_fn_to_exported_or_undefined_decls. (reader::{is_decl_die_with_undefined_symbol, load_undefined_interfaces}): Define new member functions. (build_translation_unit_and_add_to_ir): Analyze DIEs of interfaces that have undefined symbols if we were asked to load undefined interfaces. (variable_is_suppressed): Add a boolean parameter to tell if the var is decl-only. Use that decl-only flag to determine if the variable is suppressed. A non-member decl-only variable won't be suppressed if we were asked to load undefined interfaces. (build_or_get_var_decl_if_not_suppressed): Add a boolean parameter to tell if the var is decl-only. (potential_member_fn_should_be_dropped): A potential non-virtual member function with no symbol is now dropped on the floor regardless of it has a mangled name or not. (build_var_decl): If the var has an undefined symbol, then set that symbol. (build_function_decl): If the function has an undefined symbol, then set that symbol. (build_ir_node_from_die): Add a var or function with undefined symbol to the set of undefined vars or functions of the current corpus. * src/abg-fe-iface.cc (fe_iface::add_fn_to_exported_or_undefined_decls): Renamed fe_iface::maybe_add_fn_to_exported_decls into this. If the function has an undefined symbol then add the function to the set of undefined functions. (fe_iface::add_var_to_exported_or_undefined_decls): Renamed fe_iface::maybe_add_var_to_exported_decls into this. If the variable has an undefined symbol then add the variable to the set of undefined variables. * src/abg-ir.cc (elf_symbol::is_variable): Undefined symbol with type STT_NOTYPE are actually for undefined variables. (maybe_adjust_canonical_type): It's here, after type canonicalization that a member function is added to either the set of defined & exported functions, or to the set of functions with undefined symbols. * src/abg-reader.cc (build_function_decl, build_class_decl) (build_union_decl, handle_var_decl): Adjust. * src/abg-symtab-reader.cc (symtab::{lookup_undefined_function_symbol, lookup_undefined_variable_symbol, function_symbol_is_undefined, variable_symbol_is_undefined, collect_undefined_fns_and_vars_linkage_names}): Define new member functions. (symtab::symtab): Initialize the new cached_undefined_symbol_names_ data member. * src/abg-symtab-reader.h (symtab::{lookup_undefined_function_symbol, lookup_undefined_variable_symbol, function_symbol_is_undefined, variable_symbol_is_undefined, collect_undefined_fns_and_vars_linkage_names}): Declare new member functions. (symtab::{undefined_variable_linkage_names_, cached_undefined_symbol_names_}): Define new data members. (symtab::load_): Consider undefined symbol of type STT_NOTYPE as being undefined global variables. It's what I am seeing in ELF binaries. * src/abg-symtab-reader.h (symtab::{lookup_undefined_function_symbol, lookup_undefined_variable_symbol, function_symbol_is_undefined, variable_symbol_is_undefined}): Declare new member functions. (symtab::{undefined_function_linkage_names_, undefined_variable_linkage_names_}): Define new member variables. * src/abg-writer.cc (write_var_decl, write_function_decl): Emit a reference to a symbol only when the symbol is defined. * tools/abicompat.cc (report_function_changes) (report_variable_changes) (compare_expected_against_provided_functions) (compare_expected_against_provided_variables): Define new static functions. (perform_compat_check_in_weak_mode): Use the new static functions above. Compare interfaces expected by the application corpus against interfaces provided by the library. Report the changes. Do that in the reverse direction as well. (read_corpus): Instruct the corpus reader to load the set of undefined interfaces too. * tests/data/test-abicompat/test6-var-changed-app: Remove file. * tests/data/test-abicompat/test6-var-changed-app.cc: Likewise. * tests/data/test-abicompat/libtest6-undefined-var.so: Add new binary input file. * tests/data/test-abicompat/test6-undefined-var.cc: Add sourcefile for the binary input file above. * tests/data/test-abicompat/test6-var-changed-report-2.txt: New reference output file. * tests/data/Makefile.am: Update the list of distributed files accordingly. * tests/data/test-abicompat/libtest5-fn-changed-libapp-v0.so: Adjust. * tests/data/test-abicompat/libtest5-fn-changed-libapp-v1.so: Likewise. * tests/data/test-abicompat/libtest6-var-changed-libapp-v0.so: Likewise. * tests/data/test-abicompat/libtest6-var-changed-libapp-v1.so: Likewise. * tests/data/test-abicompat/libtest7-fn-changed-libapp-v0.so: Likewise. * tests/data/test-abicompat/libtest7-fn-changed-libapp-v1.so: Likewise. * tests/data/test-abicompat/libtest8-fn-changed-libapp-v0.so: Likewise. * tests/data/test-abicompat/libtest8-fn-changed-libapp-v1.so: Likewise. * tests/data/test-abicompat/libtest9-fn-changed-v0.so: Likewise. * tests/data/test-abicompat/libtest9-fn-changed-v1.so: Likewise. * tests/data/test-abicompat/test5-fn-changed-app: Likewise. * tests/data/test-abicompat/test6-var-changed-libapp-v0.cc: Likewise. * tests/data/test-abicompat/test6-var-changed-libapp-v1.cc: Likewise. * tests/data/test-abicompat/test6-var-changed-report-0.txt: Likewise. * tests/data/test-abicompat/test6-var-changed-report-1.txt: Likewise. * tests/data/test-abicompat/test7-fn-changed-app: Likewise. * tests/data/test-abicompat/test7-fn-changed-report-1.txt: Likewise. * tests/data/test-abicompat/test7-fn-changed-report-2.txt: Likewise. * tests/data/test-abicompat/test8-fn-changed-app: Likewise. * tests/data/test-abicompat/test8-fn-changed-libapp-v1.c: Likewise. * tests/data/test-abicompat/test9-fn-changed-app: Likewise. * tests/data/test-abicompat/test9-fn-changed-app.cc: Likewise. * tests/data/test-annotate/libtest23.so.abi: Likewise. * tests/data/test-annotate/libtest24-drop-fns-2.so.abi: Likewise. * tests/data/test-annotate/libtest24-drop-fns.so.abi: Likewise. * tests/data/test-annotate/test1.abi: Likewise. * tests/data/test-annotate/test14-pr18893.so.ab: Likewise.i * tests/data/test-annotate/test15-pr18892.so.abi: Likewise. * tests/data/test-annotate/test17-pr19027.so.abi: Likewise. * tests/data/test-annotate/test18-pr19037-libvtkRenderingLIC-6.1.so.abi: Likewise. * tests/data/test-annotate/test19-pr19023-libtcmalloc_and_profiler.so.abi: Likewise. * tests/data/test-annotate/test2.so.abi: Likewise. * tests/data/test-annotate/test20-pr19025-libvtkParallelCore-6.1.so.abi: Likewise. * tests/data/test-annotate/test21-pr19092.so.abi: Likewise. * tests/data/test-annotate/test8-qualified-this-pointer.so.abi: Likewise. * tests/data/test-diff-pkg/dbus-glib-0.104-3.fc23.x86_64--dbus-glib-0.104-3.fc23.armv7hl-report-0.txt: Likewise. * tests/data/test-diff-pkg/libICE-1.0.6-1.el6.x86_64.rpm--libICE-1.0.9-2.el7.x86_64.rpm-report-0.txt: Likewise. * tests/data/test-fedabipkgdiff/test0-from-fc20-to-fc23-dbus-glib-report-0.txt: Likewise. * tests/data/test-fedabipkgdiff/test2-dbus-glib-0.100.2-2.fc20--dbus-glib-0.106-1.fc23-report-0.txt: Likewise. * tests/data/test-fedabipkgdiff/test3-dbus-glib-0.100.2-2.fc20.i686--dbus-glib-0.106-1.fc23.i686-report-0.txt: Likewise. * tests/data/test-fedabipkgdiff/vte291-0.39.1-1.fc22.x86_64--vte291-0.39.90-1.fc22.x86_64-report-0.txt: Likewise. * tests/data/test-read-dwarf/PR22015-libboost_iostreams.so.abi: Likewise. * tests/data/test-read-dwarf/PR22122-libftdc.so.abi: Likewise. * tests/data/test-read-dwarf/PR26261/PR26261-exe.abi: Likewise. * tests/data/test-read-dwarf/libtest23.so.abi: Likewise. * tests/data/test-read-dwarf/libtest24-drop-fns-2.so.abi: Likewise. * tests/data/test-read-dwarf/libtest24-drop-fns.so.abi: Likewise. * tests/data/test-read-dwarf/test-libaaudio.so.abi: Likewise. * tests/data/test-read-dwarf/test-libandroid.so.abi: Likewise. * tests/data/test-read-dwarf/test1.abi: Likewise. * tests/data/test-read-dwarf/test1.hash.abi: Likewise. * tests/data/test-read-dwarf/test10-pr18818-gcc.so.abi: Likewise. * tests/data/test-read-dwarf/test11-pr18828.so.abi: Likewise. * tests/data/test-read-dwarf/test12-pr18844.so.abi: Likewise. * tests/data/test-read-dwarf/test14-pr18893.so.abi: Likewise. * tests/data/test-read-dwarf/test15-pr18892.so.abi: Likewise. * tests/data/test-read-dwarf/test16-pr18904.so.abi: Likewise. * tests/data/test-read-dwarf/test17-pr19027.so.abi: Likewise. * tests/data/test-read-dwarf/test18-pr19037-libvtkRenderingLIC-6.1.so.abi: Likewise. * tests/data/test-read-dwarf/test19-pr19023-libtcmalloc_and_profiler.so.abi: Likewise. * tests/data/test-read-dwarf/test2.so.abi: Likewise. * tests/data/test-read-dwarf/test2.so.hash.abi: Likewise. * tests/data/test-read-dwarf/test20-pr19025-libvtkParallelCore-6.1.so.abi: Likewise. * tests/data/test-read-dwarf/test21-pr19092.so.abi: Likewise. * tests/data/test-read-dwarf/test22-pr19097-libstdc++.so.6.0.17.so.abi: Likewise. * tests/data/test-read-dwarf/test8-qualified-this-pointer.so.abi: Likewise. * tests/data/test-read-dwarf/test8-qualified-this-pointer.so.hash.abi: Likewise. * tests/data/test-read-dwarf/test9-pr18818-clang.so.abi: Likewise. * tests/test-abicompat.cc (in_out_specs): Adjust. * tests/test-read-ctf.cc (test_task_ctf::perform): Do not load undefined interfaces, by default. * tests/test-symtab.cc (Symtab::SimpleSymtabs) (Symtab::SymtabWithWhitelist, Symtab::AliasedFunctionSymbols): Adjust the expected undefined variable symbols counts. Signed-off-by: Dodji Seketeli --- include/abg-corpus.h | 31 +- include/abg-fe-iface.h | 8 +- src/abg-btf-reader.cc | 4 +- src/abg-corpus-priv.h | 6 +- src/abg-corpus.cc | 165 +- src/abg-ctf-reader.cc | 6 +- src/abg-dwarf-reader.cc | 185 +- src/abg-fe-iface.cc | 43 +- src/abg-ir.cc | 24 +- src/abg-reader.cc | 10 +- src/abg-symtab-reader.cc | 115 +- src/abg-symtab-reader.h | 23 + src/abg-writer.cc | 8 +- tests/data/Makefile.am | 5 +- .../libtest5-fn-changed-libapp-v0.so | Bin 9611 -> 18112 bytes .../libtest5-fn-changed-libapp-v1.so | Bin 9587 -> 18080 bytes .../test-abicompat/libtest6-undefined-var.so | Bin 0 -> 17688 bytes .../libtest6-var-changed-libapp-v0.so | Bin 9486 -> 17976 bytes .../libtest6-var-changed-libapp-v1.so | Bin 9462 -> 17984 bytes .../libtest7-fn-changed-libapp-v0.so | Bin 9438 -> 17968 bytes .../libtest7-fn-changed-libapp-v1.so | Bin 9462 -> 17992 bytes .../libtest8-fn-changed-libapp-v0.so | Bin 8940 -> 17376 bytes .../libtest8-fn-changed-libapp-v1.so | Bin 8972 -> 17416 bytes .../test-abicompat/libtest9-fn-changed-v0.so | Bin 9004 -> 17456 bytes .../test-abicompat/libtest9-fn-changed-v1.so | Bin 9036 -> 17488 bytes .../data/test-abicompat/test5-fn-changed-app | Bin 9881 -> 26040 bytes .../test-abicompat/test6-undefined-var.cc | 12 + .../data/test-abicompat/test6-var-changed-app | Bin 9822 -> 0 bytes .../test-abicompat/test6-var-changed-app.cc | 15 - .../test6-var-changed-libapp-v0.cc | 1 + .../test6-var-changed-libapp-v1.cc | 1 + .../test6-var-changed-report-0.txt | 2 +- .../test6-var-changed-report-1.txt | 2 +- .../test6-var-changed-report-2.txt | 15 + .../data/test-abicompat/test7-fn-changed-app | Bin 9938 -> 26008 bytes .../test7-fn-changed-report-1.txt | 5 + .../test7-fn-changed-report-2.txt | 5 + .../data/test-abicompat/test8-fn-changed-app | Bin 9700 -> 25864 bytes .../test8-fn-changed-libapp-v1.c | 2 +- .../data/test-abicompat/test9-fn-changed-app | Bin 10035 -> 26008 bytes .../test-abicompat/test9-fn-changed-app.cc | 2 +- tests/data/test-annotate/libtest23.so.abi | 711 +- .../test-annotate/libtest24-drop-fns-2.so.abi | 576 - .../test-annotate/libtest24-drop-fns.so.abi | 576 - tests/data/test-annotate/test1.abi | 74 +- .../data/test-annotate/test14-pr18893.so.abi | 14336 +--- .../data/test-annotate/test15-pr18892.so.abi | 25174 ++----- .../data/test-annotate/test17-pr19027.so.abi | 18027 +---- ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi | 30166 +++----- ...19-pr19023-libtcmalloc_and_profiler.so.abi | 27423 ++----- tests/data/test-annotate/test2.so.abi | 36 - ...st20-pr19025-libvtkParallelCore-6.1.so.abi | 28876 ++------ .../data/test-annotate/test21-pr19092.so.abi | 4308 +- .../test8-qualified-this-pointer.so.abi | 21 +- ...bus-glib-0.104-3.fc23.armv7hl-report-0.txt | 7 +- ...libICE-1.0.9-2.el7.x86_64.rpm-report-0.txt | 6 +- ...0-from-fc20-to-fc23-dbus-glib-report-0.txt | 7 - ....fc20--dbus-glib-0.106-1.fc23-report-0.txt | 7 - ...--dbus-glib-0.106-1.fc23.i686-report-0.txt | 7 - ...-vte291-0.39.90-1.fc22.x86_64-report-0.txt | 13 +- .../PR22015-libboost_iostreams.so.abi | 2466 +- .../test-read-dwarf/PR22122-libftdc.so.abi | 3425 +- .../test-read-dwarf/PR26261/PR26261-exe.abi | 42 +- tests/data/test-read-dwarf/libtest23.so.abi | 411 +- .../libtest24-drop-fns-2.so.abi | 321 - .../test-read-dwarf/libtest24-drop-fns.so.abi | 321 - .../test-read-dwarf/test-libaaudio.so.abi | 1398 +- .../test-read-dwarf/test-libandroid.so.abi | 38724 ++++------ tests/data/test-read-dwarf/test1.abi | 54 +- tests/data/test-read-dwarf/test1.hash.abi | 32 - .../test-read-dwarf/test10-pr18818-gcc.so.abi | 4340 +- .../test-read-dwarf/test11-pr18828.so.abi | 8812 +-- .../test-read-dwarf/test12-pr18844.so.abi | 17781 +---- .../test-read-dwarf/test14-pr18893.so.abi | 7500 +- .../test-read-dwarf/test15-pr18892.so.abi | 20419 ++---- .../test-read-dwarf/test16-pr18904.so.abi | 6042 +- .../test-read-dwarf/test17-pr19027.so.abi | 13682 +--- ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi | 18170 ++--- ...19-pr19023-libtcmalloc_and_profiler.so.abi | 19957 ++---- tests/data/test-read-dwarf/test2.so.abi | 24 - tests/data/test-read-dwarf/test2.so.hash.abi | 24 - ...st20-pr19025-libvtkParallelCore-6.1.so.abi | 17154 +---- .../test-read-dwarf/test21-pr19092.so.abi | 3232 +- .../test22-pr19097-libstdc++.so.6.0.17.so.abi | 59856 +++++----------- .../test8-qualified-this-pointer.so.abi | 18 +- .../test8-qualified-this-pointer.so.hash.abi | 6 - .../test9-pr18818-clang.so.abi | 3052 +- tests/test-abicompat.cc | 13 +- tests/test-read-ctf.cc | 2 +- tests/test-symtab.cc | 48 +- tools/abicompat.cc | 558 +- 91 files changed, 102720 insertions(+), 296210 deletions(-) create mode 100755 tests/data/test-abicompat/libtest6-undefined-var.so create mode 100644 tests/data/test-abicompat/test6-undefined-var.cc delete mode 100755 tests/data/test-abicompat/test6-var-changed-app delete mode 100644 tests/data/test-abicompat/test6-var-changed-app.cc create mode 100644 tests/data/test-abicompat/test6-var-changed-report-2.txt [The patch is too big for the mailing list so I am attaching it in gzipped form below].