* [PATCH] Bug 26769 - Fix missing types in abixml output @ 2020-11-24 10:30 Dodji Seketeli 2021-12-22 9:54 ` 'tests/runtestslowselfcompare.sh' (was: [PATCH] Bug 26769 - Fix missing types in abixml output) Thomas Schwinge 0 siblings, 1 reply; 5+ messages in thread From: Dodji Seketeli @ 2020-11-24 10:30 UTC (permalink / raw) To: libabigail [-- Attachment #1: Type: text/plain, Size: 9398 bytes --] Hello, The symptom of the issue at hand is that sometimes there can be types missing from the abixml output. This happens when analysing some C++ code bases. The core of the issue is the following. Support we have a type "struct S" defined somewhere as: struct S // #0 { int dm1; char dm2; }; S s; Suppose that in another translation unit, we have the class 'S' being extended to add a member type to it: struct S // #1 { typedef int dm1_type; }; typedef S::dm1_type Integer; Integer something; When emitting the abixml for the codebase, the definition of the typedef S::dm1_type can be missing. Note that in location #1, struct S is considered declaration-only. It's definition is in another translation unit, in location #0. So the abixml writer emits the 'struct S' defined in location #0, but forgets to emit the 'struct S' in #1, which is indirectly used for the sole purpose of using its member type S::dm1_type. This patch emits the S::dm1_type type that is mistakenly forgotten today. Now that the "struct S" of #1 is also emitted, a tangent problem is uncovered: S in #0 can be wrongly thought to be equivalent to S in #1, for ABI purposes This is because of an ODR-based optimization that is used for C++. That is, the two struct S can be wrongly considered equivalent just because they have the same name. Note that ODR means "One Definition Rule[1]" This patch removes the ODR-based optimization and thus fixes many of the issues uncovered by the previous changes. The patch also uncovered that some non-static variables were sometimes wrongly being added to the set of exported variables, while libabigail reads corpora from abixml. The patch fixes this as well. [1]: One Definition Rule: https://en.wikipedia.org/wiki/One_Definition_Rule * include/abg-corpus.h (corpus::{record_canonical_type, lookup_canonical_type}): Remove function declarations. * src/abg-corpus-priv.h (corpus::priv::canonical_types_): Remove data member. * src/abg-corpus.cc (corpus::{record_canonical_type, lookup_canonical_type}): Remove functions. * src/abg-ir.cc (type_eligible_for_odr_based_comparison): Remove static function. (type_base::get_canonical_type_for): Don't perform the ODR-based optimization for C++ anymore. * src/abg-reader.cc (read_context&::maybe_add_var_to_exported_decls): Don't add a variable that hasn't been added to its scope. Otherwise, it means we added a variable that wasn't yet properly constructed. Also add a new overload for var_decl_sptr&. (build_var_decl): Do not add the var to its the set of exported declaration before we are sure it has been fully constructed and added to the scope it belongs. (build_class_decl): Only add *static* data members to the list of exported declarations. (handle_var_decl): A var decl seen here is a global variable declaration. Add it to the list of exported declarations. * src/abg-writer.cc (write_context::decl_only_type_is_emitted): Constify parameter. (write_translation_unit): Do not forget to emit referenced types that were maybe not canonicalized. Also, avoid using noop_deleter when it's not necessary. (write_namespace_decl): Do not forget to emit canonicalized types that are present in namespaces other than the global namespace. * tests/runtestslowselfcompare.sh.in: New test that compares libabigail.so against its own ABIXML representation. * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to source distribution. This test is too slow to be run during the course of 'make check'. It takes more than 5 minutes on my slow box here. Rather, it can be run using 'make check-self-compare'. I plan to run this before releases now. * tests/data/test-annotate/libtest24-drop-fns-2.so.abi: Adjust. * tests/data/test-annotate/libtest24-drop-fns.so.abi: Likewise. * tests/data/test-annotate/test0.abi: Likewise. * tests/data/test-annotate/test13-pr18894.so.abi: Likewise. * tests/data/test-annotate/test14-pr18893.so.abi: Likewise. * tests/data/test-annotate/test15-pr18892.so.abi: Likewise. * tests/data/test-annotate/test17-pr19027.so.abi: Likewise. * tests/data/test-annotate/test18-pr19037-libvtkRenderingLIC-6.1.so.abi: Likewise. * tests/data/test-annotate/test19-pr19023-libtcmalloc_and_profiler.so.abi: Likewise. * tests/data/test-annotate/test20-pr19025-libvtkParallelCore-6.1.so.abi: Likewise. * tests/data/test-annotate/test21-pr19092.so.abi: Likewise. * tests/data/test-read-dwarf/PR22015-libboost_iostreams.so.abi: Likewise. * tests/data/test-read-dwarf/PR22122-libftdc.so.abi: Likewise. * tests/data/test-read-dwarf/PR25042-libgdbm-clang-dwarf5.so.6.0.0.abi: Likewise. * tests/data/test-read-dwarf/PR26261/PR26261-exe.abi: Likewise. * tests/data/test-read-dwarf/libtest24-drop-fns-2.so.abi: Likewise. * tests/data/test-read-dwarf/libtest24-drop-fns.so.abi: Likewise. * tests/data/test-read-dwarf/test-libandroid.so.abi: Likewise. * tests/data/test-read-dwarf/test0.abi: Likewise. * tests/data/test-read-dwarf/test0.hash.abi: Likewise. * tests/data/test-read-dwarf/test10-pr18818-gcc.so.abi: Likewise. * tests/data/test-read-dwarf/test11-pr18828.so.abi: Likewise. * tests/data/test-read-dwarf/test12-pr18844.so.abi: Likewise. * tests/data/test-read-dwarf/test14-pr18893.so.abi: Likewise. * tests/data/test-read-dwarf/test15-pr18892.so.abi: Likewise. * tests/data/test-read-dwarf/test16-pr18904.so.abi: Likewise. * tests/data/test-read-dwarf/test17-pr19027.so.abi: Likewise. * tests/data/test-read-dwarf/test18-pr19037-libvtkRenderingLIC-6.1.so.abi: Likewise. * tests/data/test-read-dwarf/test19-pr19023-libtcmalloc_and_profiler.so.abi: Likewise. * tests/data/test-read-dwarf/test20-pr19025-libvtkParallelCore-6.1.so.abi: Likewise. * tests/data/test-read-dwarf/test21-pr19092.so.abi: Likewise. * tests/data/test-read-dwarf/test22-pr19097-libstdc++.so.6.0.17.so.abi: Likewise. * tests/data/test-read-dwarf/test9-pr18818-clang.so.abi: Likewise. * tests/data/test-read-write/test28-without-std-fns-ref.xml: Likewise. * tests/data/test-read-write/test28-without-std-vars-ref.xml: Likewise. * tests/data/test-read-write/test6.xml: Likewise. Signed-off-by: Dodji Seketeli <dodji@redhat.com> --- Makefile.am | 6 + configure.ac | 2 + include/abg-corpus.h | 2 - src/abg-corpus-priv.h | 1 - src/abg-corpus.cc | 40 - src/abg-ir.cc | 113 +- src/abg-reader.cc | 30 +- src/abg-writer.cc | 72 +- tests/Makefile.am | 15 +- .../data/test-annotate/libtest24-drop-fns-2.so.abi | 38 +- tests/data/test-annotate/libtest24-drop-fns.so.abi | 38 +- tests/data/test-annotate/test0.abi | 16 +- tests/data/test-annotate/test14-pr18893.so.abi | 14712 ++- tests/data/test-annotate/test15-pr18892.so.abi | 43434 ++++--- tests/data/test-annotate/test17-pr19027.so.abi | 69043 +++++++---- .../test18-pr19037-libvtkRenderingLIC-6.1.so.abi | 13622 ++- .../test19-pr19023-libtcmalloc_and_profiler.so.abi | 36146 +++--- .../test20-pr19025-libvtkParallelCore-6.1.so.abi | 37504 +++--- tests/data/test-annotate/test21-pr19092.so.abi | 6712 +- .../PR22015-libboost_iostreams.so.abi | 2273 +- tests/data/test-read-dwarf/PR22122-libftdc.so.abi | 1025 +- .../PR25042-libgdbm-clang-dwarf5.so.6.0.0.abi | 811 +- tests/data/test-read-dwarf/PR26261/PR26261-exe.abi | 30 +- .../test-read-dwarf/libtest24-drop-fns-2.so.abi | 20 +- .../data/test-read-dwarf/libtest24-drop-fns.so.abi | 8 +- tests/data/test-read-dwarf/test-libandroid.so.abi | 110042 ++++++++++-------- tests/data/test-read-dwarf/test0.abi | 12 +- tests/data/test-read-dwarf/test0.hash.abi | 12 +- .../data/test-read-dwarf/test10-pr18818-gcc.so.abi | 6983 +- tests/data/test-read-dwarf/test11-pr18828.so.abi | 53964 ++++----- tests/data/test-read-dwarf/test12-pr18844.so.abi | 61126 +++++----- tests/data/test-read-dwarf/test14-pr18893.so.abi | 11252 +- tests/data/test-read-dwarf/test15-pr18892.so.abi | 34025 +++--- tests/data/test-read-dwarf/test16-pr18904.so.abi | 59148 +++++----- tests/data/test-read-dwarf/test17-pr19027.so.abi | 38119 ++++-- .../test18-pr19037-libvtkRenderingLIC-6.1.so.abi | 9447 +- .../test19-pr19023-libtcmalloc_and_profiler.so.abi | 28340 +++-- .../test20-pr19025-libvtkParallelCore-6.1.so.abi | 26872 +++-- tests/data/test-read-dwarf/test21-pr19092.so.abi | 5950 +- .../test22-pr19097-libstdc++.so.6.0.17.so.abi | 68467 +++++++---- .../test-read-dwarf/test9-pr18818-clang.so.abi | 4830 +- .../test-read-write/test28-without-std-fns-ref.xml | 26 +- .../test28-without-std-vars-ref.xml | 42 +- tests/data/test-read-write/test6.xml | 6 +- tests/runtestslowselfcompare.sh.in | 13 + 45 files changed, 444887 insertions(+), 299502 deletions(-) create mode 100644 tests/runtestslowselfcompare.sh.in I am attaching the gzipped form of this patch as it's too big otherwise. [-- Attachment #2: gzipped version of the patch --] [-- Type: application/gzip, Size: 4524737 bytes --] [-- Attachment #3: Type: text/plain, Size: 13 bytes --] -- Dodji ^ permalink raw reply [flat|nested] 5+ messages in thread
* 'tests/runtestslowselfcompare.sh' (was: [PATCH] Bug 26769 - Fix missing types in abixml output) 2020-11-24 10:30 [PATCH] Bug 26769 - Fix missing types in abixml output Dodji Seketeli @ 2021-12-22 9:54 ` Thomas Schwinge 2021-12-22 10:49 ` [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' Thomas Schwinge 0 siblings, 1 reply; 5+ messages in thread From: Thomas Schwinge @ 2021-12-22 9:54 UTC (permalink / raw) To: Dodji Seketeli, libabigail Hi! On 2020-11-24T11:30:02+0100, Dodji Seketeli via Libabigail <libabigail@sourceware.org> wrote: > * tests/runtestslowselfcompare.sh.in: New test that compares > libabigail.so against its own ABIXML representation. I consider this to be a pretty important test case -- "eat our own dog food". Thus, I find this a bit unfortunate: > * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to > source distribution. This test is too slow to be run during the > course of 'make check'. It takes more than 5 minutes on my slow > box here. Rather, it can be run using 'make check-self-compare'. > I plan to run this before releases now. ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during standard 'make check'. Or may we can change that now? Are you still seeing that "It takes more than 5 minutes on my slow box here"? Because, on my eight years old Dell Precision M4700, I see: $ \time make check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes 20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468maxresident)k 20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984maxresident)k 20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016maxresident)k So, ~21 s. All the other test cases, running in parallel (just '-j5'): $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=no' 364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568maxresident)k 359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720maxresident)k 359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292maxresident)k So, ~61 s. Additionally running 'tests/runtestslowselfcompare.sh': $ \time make check -j5 ENABLE_SLOW_TEST=yes 389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036maxresident)k 387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000maxresident)k 388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356maxresident)k So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the 'make check -j5' take just ~5 s longer -- acceptable, in my opinion. Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030 "CONTRIBUTING: Update instructions about regression tests", it was noted that... | This is an important regression test. The | problem is that it can takes twice as much time as make distcheck. So | we've put it into its own separate target. Given the "5 minutes" number from above, this comment means that a 'make distcheck' (or rather 'make distcheck-fast', I suppose?) would run ~2.5 min? I've got the following numbers: $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=no' 935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144maxresident)k 946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372maxresident)k 935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388maxresident)k So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone takes just 1/15 the time of that, not "twice as much". Additionally enabling 'tests/runtestslowselfcompare.sh' here: $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=yes 965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760maxresident)k 981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836maxresident)k 972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224maxresident)k ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s longer -- acceptable, in my opinion. Therefore I propose to do away with 'ENABLE_SLOW_TEST', and merge 'make check-self-compare' into standard 'make check'. I'll be happy to prepare a patch, if we arrive at consensus, but first: happy to hear any other opinions/data. Grüße Thomas > --- a/Makefile.am > +++ b/Makefile.am > @@ -88,3 +88,9 @@ tag-and-all: distcheck > distcheck-fast: > $(MAKE) distcheck GZIP_ENV="--fast" > > +# This makes us compare libabigail.so against its own ABIXML > +# representation. It's super slow (more than 5 minutes on one of my > +# old boxes) so I do this only before releases. > +check-self-compare: > + $(MAKE) -C tests check-self-compare > --- a/tests/Makefile.am > +++ b/tests/Makefile.am > [...] > +check-self-compare: > + $(MAKE) check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes > --- /dev/null > +++ b/tests/runtestslowselfcompare.sh.in > @@ -0,0 +1,13 @@ > +#!/bin/sh -x > + > +# This program launches "abidw --abidiff src/.libs/libabigail.so" > +# This compares libabigail.so against its own ABIXML output. > +abidw=@top_builddir@/tools/abidw > +objdir=@top_builddir@/src/.libs > + > +echo "ENABLE_SLOW_TEST="$ENABLE_SLOW_TEST > + > +if test x$ENABLE_SLOW_TEST != x; then > + `$abidw --abidiff $objdir/libabigail.so` > +fi ----------------- Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' 2021-12-22 9:54 ` 'tests/runtestslowselfcompare.sh' (was: [PATCH] Bug 26769 - Fix missing types in abixml output) Thomas Schwinge @ 2021-12-22 10:49 ` Thomas Schwinge 2022-01-04 14:53 ` Dodji Seketeli 0 siblings, 1 reply; 5+ messages in thread From: Thomas Schwinge @ 2021-12-22 10:49 UTC (permalink / raw) To: libabigail; +Cc: Dodji Seketeli, Thomas Schwinge Per commit cac59a176a0c0d6d6c693cb1cfb475517ec33e97 "Bug 26769 - Fix missing types in abixml output": | * tests/runtestslowselfcompare.sh.in: New test that compares | libabigail.so against its own ABIXML representation. I consider this to be a pretty important test case -- "eat our own dog food". Thus, I find this a bit unfortunate: | * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to | source distribution. This test is too slow to be run during the | course of 'make check'. It takes more than 5 minutes on my slow | box here. Rather, it can be run using 'make check-self-compare'. | I plan to run this before releases now. ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during standard 'make check'. On my eight years old Dell Precision M4700, I see: $ \time make check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes 20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468maxresident)k 20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984maxresident)k 20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016maxresident)k So, ~21 s. All the other test cases, running in parallel (just '-j5'): $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=no' 364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568maxresident)k 359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720maxresident)k 359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292maxresident)k So, ~61 s. Additionally running 'tests/runtestslowselfcompare.sh': $ \time make check -j5 ENABLE_SLOW_TEST=yes 389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036maxresident)k 387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000maxresident)k 388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356maxresident)k So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the 'make check -j5' take just ~5 s longer -- acceptable, in my opinion. Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030 "CONTRIBUTING: Update instructions about regression tests", it was noted that... | This is an important regression test. The | problem is that it can takes twice as much time as make distcheck. So | we've put it into its own separate target. Given the "5 minutes" number from above, this comment means that a 'make distcheck' (or rather 'make distcheck-fast', I suppose?) would run ~2.5 min? I've got the following numbers: $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=no' 935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144maxresident)k 946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372maxresident)k 935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388maxresident)k So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone takes just 1/15 the time of that, not "twice as much". Additionally enabling 'tests/runtestslowselfcompare.sh' here: $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=yes 965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760maxresident)k 981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836maxresident)k 972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224maxresident)k ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s longer -- acceptable, in my opinion. * tests/runtestslowselfcompare.sh.in: Rename to... * tests/runtestselfcompare.sh.in: ... this, and assume 'ENABLE_SLOW_TEST' set. * tests/Makefile.am: Adjust. (check-self-compare): Remove target. * Makefile.am: Likewise. * configure.ac: Adjust. * CONTRIBUTING: Likewise. --- CONTRIBUTING | 18 ++++++++---------- Makefile.am | 9 +-------- configure.ac | 4 ++-- tests/Makefile.am | 14 ++++---------- ...fcompare.sh.in => runtestselfcompare.sh.in} | 7 +------ 5 files changed, 16 insertions(+), 36 deletions(-) rename tests/{runtestslowselfcompare.sh.in => runtestselfcompare.sh.in} (61%) diff --git a/CONTRIBUTING b/CONTRIBUTING index 5ab7bcce..588f87dd 100644 --- a/CONTRIBUTING +++ b/CONTRIBUTING @@ -58,6 +58,9 @@ tests in parallel to make whole thing go faster by doing: make -jN -lN check +distcheck +--------- + If you want to test the fabrication of the distribution tarball (this is important, because that is how we do to actually release the tarball of the project that you can download from the internet) then @@ -85,19 +88,14 @@ Please make sure you always launch "make distcheck-fast" before sending a patch, so that you are sure that we can always build a tarball after your patch is applied to the source tree. -A complementary regression checking target is "check-self-compare". -You invoke it by doing "make check-self-compare". That target -analyzes the ABI of the libabigail.so shared object, serializes it +tests/runtestselfcompare.sh +--------------------------- + +This one analyzes the ABI of the libabigail.so shared object, serializes it into the ABIXML format and then compares the ABI internal representation gathered from the libabigail.so binary against the one gathered from the ABIXML format. The two should be equal if -everything goes right. This is an important regression test. The -problem is that it can take a considerable amount of time. -So we've put it into its own separate target. - -So, to be complete the regression checking command to run against your -patch should be: "make check-self-compare distcheck-fast -j16", if you have -a machine with a 16 threads processors, for instance. +everything goes right. This is an important regression test. Launching regression tests in Valgrind -------------------------------------- diff --git a/Makefile.am b/Makefile.am index 65a1b121..99353e82 100644 --- a/Makefile.am +++ b/Makefile.am @@ -80,17 +80,10 @@ tarball-and-tag-and-upload: tarball-and-all tarball-and-all: tag-and-all $(MAKE) tarball -tag-and-all: distcheck check-self-compare +tag-and-all: distcheck $(MAKE) tag-release-only # the default for GZIP_ENV is --best, which is pretty slow for check runs # distcheck-fast therefore compresses with --fast instead distcheck-fast: $(MAKE) distcheck GZIP_ENV="--fast" - -# This makes us compare libabigail.so against its own ABIXML -# representation. It's super slow (more than 5 minutes on one of my -# old boxes) so I do this only before releases. -check-self-compare: - $(MAKE) -C tests check-self-compare - diff --git a/configure.ac b/configure.ac index 29130175..1172a9fe 100644 --- a/configure.ac +++ b/configure.ac @@ -969,8 +969,8 @@ libabigail.pc dnl Some test scripts are generated by autofoo. AC_CONFIG_FILES([tests/runtestcanonicalizetypes.sh], [chmod +x tests/runtestcanonicalizetypes.sh]) - AC_CONFIG_FILES([tests/runtestslowselfcompare.sh], - [chmod +x tests/runtestslowselfcompare.sh]) +AC_CONFIG_FILES([tests/runtestselfcompare.sh], + [chmod +x tests/runtestselfcompare.sh]) AC_CONFIG_FILES([tests/mockfedabipkgdiff], [chmod +x tests/mockfedabipkgdiff]) AC_CONFIG_FILES([tests/runtestfedabipkgdiff.py], diff --git a/tests/Makefile.am b/tests/Makefile.am index 17e2d4b4..27ecb623 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -12,11 +12,9 @@ FEDABIPKGDIFF_TEST += runtestfedabipkgdiff.py endif endif -RUN_TEST_SLOW_SELF_COMPARE=runtestslowselfcompare.sh - # rather expensive tests (keep in this order), > 1s runtime TESTS= \ -$(RUN_TEST_SLOW_SELF_COMPARE) \ +runtestselfcompare.sh \ runtestdiffsuppr \ runtesttypesstability \ runtestdiffpkg \ @@ -60,7 +58,7 @@ endif EXTRA_DIST = \ runtestcanonicalizetypes.sh.in \ runtestfedabipkgdiff.py.in \ -runtestslowselfcompare.sh.in \ +runtestselfcompare.sh.in \ mockfedabipkgdiff.in \ test-valgrind-suppressions.supp @@ -183,8 +181,8 @@ testdiff2_LDADD=$(top_builddir)/src/libabigail.la printdifftree_SOURCES = print-diff-tree.cc printdifftree_LDADD = $(top_builddir)/src/libabigail.la -runtestslowselfcompare_sh_SOURCES = -runtestslowselfcompare.sh$(EXEEXT): +runtestselfcompare_sh_SOURCES = +runtestselfcompare.sh$(EXEEXT): runtestcanonicalizetypes_sh_SOURCES = runtestcanonicalizetypes.sh$(EXEEXT): @@ -242,7 +240,3 @@ check-valgrind-memcheck-recursive: check-valgrind-helgrind-recursive: $(MAKE) check-valgrind-helgrind VALGRIND_FLAGS="${RECURSIVE_VALGRIND_FLAGS}" - -check-self-compare: - $(MAKE) check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes - diff --git a/tests/runtestslowselfcompare.sh.in b/tests/runtestselfcompare.sh.in similarity index 61% rename from tests/runtestslowselfcompare.sh.in rename to tests/runtestselfcompare.sh.in index 3d0ef123..a6f16009 100644 --- a/tests/runtestslowselfcompare.sh.in +++ b/tests/runtestselfcompare.sh.in @@ -5,9 +5,4 @@ abidw=@top_builddir@/tools/abidw objdir=@top_builddir@/src/.libs -echo "ENABLE_SLOW_TEST="$ENABLE_SLOW_TEST - -if test x$ENABLE_SLOW_TEST != x; then - `$abidw --abidiff $objdir/libabigail.so` -fi - +`$abidw --abidiff $objdir/libabigail.so` -- 2.25.1 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' 2021-12-22 10:49 ` [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' Thomas Schwinge @ 2022-01-04 14:53 ` Dodji Seketeli 2022-01-05 16:10 ` Thomas Schwinge 0 siblings, 1 reply; 5+ messages in thread From: Dodji Seketeli @ 2022-01-04 14:53 UTC (permalink / raw) To: Thomas Schwinge; +Cc: libabigail, Dodji Seketeli Hello Thomas, Thomas Schwinge <thomas@codesourcery.com> a écrit: > Per commit cac59a176a0c0d6d6c693cb1cfb475517ec33e97 > "Bug 26769 - Fix missing types in abixml output": > > | * tests/runtestslowselfcompare.sh.in: New test that compares > | libabigail.so against its own ABIXML representation. > > I consider this to be a pretty important test case -- "eat our own dog > food". > > Thus, I find this a bit unfortunate: > > | * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to > | source distribution. This test is too slow to be run during the > | course of 'make check'. It takes more than 5 minutes on my slow > | box here. Rather, it can be run using 'make check-self-compare'. > | I plan to run this before releases now. > > ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during > standard 'make check'. > > On my eight years old Dell Precision M4700, I see: > > $ \time make check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes > > 20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468maxresident)k > 20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984maxresident)k > 20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016maxresident)k > > So, ~21 s. On my machine (AMD FX8350 box) it's really still much longer than that, as you pointed above. And I know users who are in the same case, still. Yeah, surprising, I know. Your box is more than 10 times faster than mine, it seems. > All the other test cases, running in parallel (just '-j5'): > > $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=no' > > 364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568maxresident)k > 359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720maxresident)k > 359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292maxresident)k > > So, ~61 s. Additionally running 'tests/runtestslowselfcompare.sh': > > $ \time make check -j5 ENABLE_SLOW_TEST=yes > > 389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036maxresident)k > 387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000maxresident)k > 388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356maxresident)k > > So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the > 'make check -j5' take just ~5 s longer -- acceptable, in my opinion. > > Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030 > "CONTRIBUTING: Update instructions about regression tests", it was noted > that... > > | This is an important regression test. The > | problem is that it can takes twice as much time as make distcheck. So > | we've put it into its own separate target. > > Given the "5 minutes" number from above, this comment means that a > 'make distcheck' (or rather 'make distcheck-fast', I suppose?) Well, you are maybe reading too much into that sentence :-) Since that time, make distcheck grew slower. So I guess the test now takes roughly the same time as make distcheck or make distcheck-fast. > would run > ~2.5 min? I've got the following numbers: > > $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=no' > > 935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144maxresident)k > 946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372maxresident)k > 935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388maxresident)k > > So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone > takes just 1/15 the time of that, not "twice as much". > > Additionally enabling 'tests/runtestslowselfcompare.sh' here: > > $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=yes > > 965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760maxresident)k > 981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836maxresident)k > 972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224maxresident)k > > ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s > longer -- acceptable, in my opinion. On my machine, your patch "make check" takes 5 minutes longer than without. So I am reluctant to apply it. In practise, it's not a big deal, I think, as I run 'make check-self-compare' very regularly. As bizarre as it seems, the tendency I'd like us to move towards is less tests "by default". That is, split out the binaries that we have in the tarball today and keep a very minimal set of tests. The tarball is too big as it is, and yet, there are tons of tests that I'd like to run that are not present there. That is why I'd like us to progress towards having much more tests that are in a separated "test project", somehow. In that project, we'd either have binaries locally present or references to binaries (like distros packages over the interweb) to grab and run comparisons on. So I would not spent to much time on these tests that are locally present in the tarball. Rather, if you are interested in this super important testing strategy project, we could discuss it a bit more in depth. I hope this makes sense. Thank you for showing interesting in this area. [...] Cheers, -- Dodji ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' 2022-01-04 14:53 ` Dodji Seketeli @ 2022-01-05 16:10 ` Thomas Schwinge 0 siblings, 0 replies; 5+ messages in thread From: Thomas Schwinge @ 2022-01-05 16:10 UTC (permalink / raw) To: Dodji Seketeli; +Cc: libabigail Hi Dodji! Happy New Year to y'all, too! On 2022-01-04T15:53:57+0100, Dodji Seketeli <dodji@redhat.com> wrote: > Thomas Schwinge <thomas@codesourcery.com> a écrit: > >> Per commit cac59a176a0c0d6d6c693cb1cfb475517ec33e97 >> "Bug 26769 - Fix missing types in abixml output": >> >> | * tests/runtestslowselfcompare.sh.in: New test that compares >> | libabigail.so against its own ABIXML representation. >> >> I consider this to be a pretty important test case -- "eat our own dog >> food". >> >> Thus, I find this a bit unfortunate: >> >> | * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to >> | source distribution. This test is too slow to be run during the >> | course of 'make check'. It takes more than 5 minutes on my slow >> | box here. Rather, it can be run using 'make check-self-compare'. >> | I plan to run this before releases now. >> >> ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during >> standard 'make check'. >> >> On my eight years old Dell Precision M4700, I see: >> >> $ \time make check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes >> >> 20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468maxresident)k >> 20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984maxresident)k >> 20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016maxresident)k >> >> So, ~21 s. > > On my machine (AMD FX8350 box) it's really still much longer than that, as > you pointed above. And I know users who are in the same case, still. > Yeah, surprising, I know. Your box is more than 10 times faster than > mine, it seems. "Interesting" ;-) -- thanks for confirming your numbers. Per a quick web search, your AMD FX8350 would be just a little older than my Dell Precision M4700 with "Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz", so I wonder where the rather big difference is coming from. Mine has 24 GiB of RAM, and I'm caching the "WDC WD7500BPKT-7" HDD with a "KINGSTON SA400S3" SSD; maybe that's it. >> All the other test cases, running in parallel (just '-j5'): >> >> $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=no' >> >> 364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568maxresident)k >> 359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720maxresident)k >> 359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292maxresident)k >> >> So, ~61 s. Additionally running 'tests/runtestslowselfcompare.sh': >> >> $ \time make check -j5 ENABLE_SLOW_TEST=yes >> >> 389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036maxresident)k >> 387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000maxresident)k >> 388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356maxresident)k >> >> So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the >> 'make check -j5' take just ~5 s longer -- acceptable, in my opinion. >> >> Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030 >> "CONTRIBUTING: Update instructions about regression tests", it was noted >> that... >> >> | This is an important regression test. The >> | problem is that it can takes twice as much time as make distcheck. So >> | we've put it into its own separate target. >> >> Given the "5 minutes" number from above, this comment means that a >> 'make distcheck' (or rather 'make distcheck-fast', I suppose?) > > Well, you are maybe reading too much into that sentence :-) Since that time, > make distcheck grew slower. So I guess the test now takes roughly the > same time as make distcheck or make distcheck-fast. > >> would run >> ~2.5 min? I've got the following numbers: >> >> $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=no' >> >> 935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144maxresident)k >> 946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372maxresident)k >> 935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388maxresident)k >> >> So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone >> takes just 1/15 the time of that, not "twice as much". >> >> Additionally enabling 'tests/runtestslowselfcompare.sh' here: >> >> $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=yes >> >> 965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760maxresident)k >> 981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836maxresident)k >> 972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224maxresident)k >> >> ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s >> longer -- acceptable, in my opinion. > > On my machine, your patch "make check" takes 5 minutes longer than > without. So I am reluctant to apply it. Sure, understood, and no worries. > In practise, it's not a big deal, I think, as I run 'make > check-self-compare' very regularly. > > As bizarre as it seems, the tendency I'd like us to move towards is less > tests "by default". That is, split out the binaries that we have in the > tarball today and keep a very minimal set of tests. My goal is just to establish a testing baseline, so that I can be reasonably sure that changes I may be doing don't regress anything. For the time being, I might just locally set 'ENABLE_SLOW_TEST=yes'; and at least we've now got some numbers (confirmed/new) in the archives. > The tarball is too > big as it is, and yet, there are tons of tests that I'd like to run that > are not present there. > > That is why I'd like us to progress towards having much more tests that > are in a separated "test project", somehow. In that project, we'd > either have binaries locally present or references to binaries (like > distros packages over the interweb) to grab and run comparisons on. Yes, that makes sense. (I did wonder about the Git repository/checkout size as well as the fact that huge binary blobs are stored there without sources reference.) > So I would not spent to much time on these tests that are locally > present in the tarball. Rather, if you are interested in this super > important testing strategy project, we could discuss it a bit more in > depth. Heh, not at this time, sorry. ;-) Grüße Thomas ----------------- Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955 ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-01-05 16:10 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-11-24 10:30 [PATCH] Bug 26769 - Fix missing types in abixml output Dodji Seketeli 2021-12-22 9:54 ` 'tests/runtestslowselfcompare.sh' (was: [PATCH] Bug 26769 - Fix missing types in abixml output) Thomas Schwinge 2021-12-22 10:49 ` [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' Thomas Schwinge 2022-01-04 14:53 ` Dodji Seketeli 2022-01-05 16:10 ` Thomas Schwinge
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).