From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id C4115395A440 for ; Tue, 3 May 2022 19:08:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C4115395A440 Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-327-DW5T1222Ma2QbOqd1bi7pA-1; Tue, 03 May 2022 15:08:53 -0400 X-MC-Unique: DW5T1222Ma2QbOqd1bi7pA-1 Received: by mail-oo1-f72.google.com with SMTP id c1-20020a4aa4c1000000b0035e95332a48so10382101oom.7 for ; Tue, 03 May 2022 12:08:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=/lvbKo+VxNyubhnKoTsg1fNUG9zFidl0Zu/5sVZtA+4=; b=4g8LZOqrnW2Bpqz2Zcp2vxib2eBFUTa3oTWN4gCZbnH2z2WkdjTm9Zlnan/tIY5fxM SLeJlxQheuOCzLOaDlBAKfR8bsqMurxR9tiC4cNHZ7iJBxPj/niw16x5UmNCUY+0v3qE qJV7mk+Y1fyEHMcLZp/La+PM93Y0n7wxrPQfFoe3zrBPHn4YtoWZeYL+eZvNLlZ/Xq99 P8XESUIlnki0rTc961eTPW4rnLA0RKYcIgvfe8thOW1dsPBm42r4Yk/kjAPGaUgGOvP/ S4nomVAvUB5BnlWE9aC9v4nTSk0RDSorY2odUbYouRwvUQN7vBrmGzoT1zZgSn55Y8tT NZuA== X-Gm-Message-State: AOAM533WwVYZoQn7sGUpcqt/By0FtuDBkdVskN1Ry2GOX4DeLjP7hfqn yjU+v6CH847kPAqEbMvgUvTdsKq834lDxi+4kTJsP6Oh+x/eWC+T80nos3hAMUH0ZeHMhenFJNO 8V57uog//9MloH1Mh+G8an6uhHcpvQEVH0+dIcBmtfO05EIermk+ERh12FDjTbpK8bpxpPWU= X-Received: by 2002:a05:6808:21a3:b0:325:7d2f:87b2 with SMTP id be35-20020a05680821a300b003257d2f87b2mr2542404oib.102.1651604932408; Tue, 03 May 2022 12:08:52 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwqlBvlBa9ilhklzSIyNYtp4JdyENWSVeVBtkW7nxHv+mZkHOzDKgoC1l/kA88wvm1XS/KPbw== X-Received: by 2002:a05:6808:21a3:b0:325:7d2f:87b2 with SMTP id be35-20020a05680821a300b003257d2f87b2mr2542384oib.102.1651604931700; Tue, 03 May 2022 12:08:51 -0700 (PDT) Received: from smtpclient.apple ([47.208.199.57]) by smtp.gmail.com with ESMTPSA id q8-20020a0568080ec800b00325cda1ffa3sm3709748oiv.34.2022.05.03.12.08.50 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 May 2022 12:08:51 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.80.82.1.1\)) Subject: Re: Testing Setup - More Tests and Automation? From: Ben Woodard In-Reply-To: <20220429211304.GC7305@gnu.wildebeest.org> Date: Tue, 3 May 2022 12:08:48 -0700 Cc: Mark Wielaard Message-Id: <9E390F20-185A-4F9D-A66B-3740BB860AF4@redhat.com> References: <34E58964-E930-4DF9-87CD-18D4C63DBCEB@redhat.com> <20220429211304.GC7305@gnu.wildebeest.org> To: Ben Woodard via Libabigail X-Mailer: Apple Mail (2.3696.80.82.1.1) X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-11.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2022 19:08:57 -0000 > On Apr 29, 2022, at 2:13 PM, Mark Wielaard wrote: >=20 > That said, if there are other tests, environment variables or > configure flags that could be used to catch more issues we could > enable them (on one or more builders). As long as it doesn't take more > than 10 minutes to run the extra tests on a particular builder. I don=E2=80=99t know of any specific configure flags or environmental vari= ables which could be added. I believe that some additional regression tests= are needed.=20 I have a WIP patch that I haven=E2=80=99t gotten working yet. It amounts to= : diff --git a/tests/runtestfedabipkgdiff.py.in b/tests/runtestfedabipkgdiff.= py.in index dd4e041d..3ce6878d 100755 --- a/tests/runtestfedabipkgdiff.py.in +++ b/tests/runtestfedabipkgdiff.py.in @@ -80,6 +80,35 @@ FEDABIPKGDIFF_TEST_SPECS =3D [ (['--self-compare', '-a', '--from', 'fc23', 'dbus-glib'], 'data/test-fedabipkgdiff/test7-self-compare-from-fc23-dbus-glib-repor= t-0.txt', 'output/test-fedabipkgdiff/test7-self-compare-from-fc23-dbus-glib-rep= ort-0.txt'), + + (['--self-compare', '-a', '--from', 'fc36', 'aspell'], + 'data/test-fedabipkgdiff/test8-self-compare-from-fc36-aspell-report-0= .txt', + 'output/test-fedabipkgdiff/test8-self-compare-from-fc36-aspell-report= -0.txt'), + + (['--self-compare', '-a', '--from', 'fc36', 'dyninst'], + 'data/test-fedabipkgdiff/test9-self-compare-from-fc36-dyninst-report-= 0.txt', + 'output/test-fedabipkgdiff/test9-self-compare-from-fc36-dyninst-repor= t-0.txt'), + + (['--self-compare', '-a', '--from', 'fc36', 'libabigail'], + 'data/test-fedabipkgdiff/test10-self-compare-from-fc36-libabigail-rep= ort-0.txt', + 'output/test-fedabipkgdiff/test10-self-compare-from-fc36-libabigail-r= eport-0.txt'), + + (['--self-compare', '-a', '--from', 'fc36', 'hdf5'], + 'data/test-fedabipkgdiff/test11-self-compare-from-fc36-hdf5-report-0.= txt', + 'output/test-fedabipkgdiff/test11-self-compare-from-fc36-hdf5-report-= 0.txt'), + + (['--self-compare', '-a', '--from', 'fc36', 'gcc'], + 'data/test-fedabipkgdiff/test12-self-compare-from-fc36-gcc-report-0.t= xt', + 'output/test-fedabipkgdiff/test12-self-compare-from-fc36-gcc-report-0= .txt'), + + (['--self-compare', '-a', '--from', 'fc36', 'openmpi'], + 'data/test-fedabipkgdiff/test13-self-compare-from-fc36-openmpi-report= -0.txt', + 'output/test-fedabipkgdiff/test13-self-compare-from-fc36-openmpi-repo= rt-0.txt'), + + (['--self-compare', '-a', '--from', 'fc36', 'protobuf'], + 'data/test-fedabipkgdiff/test14-self-compare-from-fc36-protobuf-repor= t-0.txt', + 'output/test-fedabipkgdiff/test14-self-compare-from-fc36-protobuf-rep= ort-0.txt'), + ] Along with the expected results in the data directory. These are the ones w= hich most frequently seem to break. Other than getting the patch to actuall= y work, I believe that for it to be acceptable to Dodji it would need to be= lumped under the slow tests and I haven=E2=80=99t figured out how to do th= at yet. Python is not my best language. So if we could have one of the builders run something like: for i in aspell dyninst libabigail hdf5 gcc openmpi proj protobuf; do fedabipkgdiff --self-compare -a --from fc36 $i >/dev/null if [ $? !=3D 0 ];then exit 1 fi done Then I think that we=E2=80=99d catch many of the problems that I have been = seeing. Notes: - Having libabigail in the list may be redundant=20 - GCC is in there because libstdc++ tends to break things. I haven=E2=80=99= t seen a problem with gcc itself. - When sharing the diff, I realized that I forgot to add =E2=80=98proj=E2= =80=99 - Note that these are all C++. libabigail rarely fails on C these days. How= ever, the HPC apps that I care most about are all C++ and Fortran. -ben