From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id CCD233858D28 for ; Thu, 28 Apr 2022 00:55:12 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org CCD233858D28 Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-252-47sfcb1NPBml19LesRl_nQ-1; Wed, 27 Apr 2022 20:55:11 -0400 X-MC-Unique: 47sfcb1NPBml19LesRl_nQ-1 Received: by mail-qv1-f69.google.com with SMTP id jx10-20020a0562142b0a00b00456531ba83dso2039297qvb.16 for ; Wed, 27 Apr 2022 17:55:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:cc:to:message-id; bh=0obdwZyDS+Ak6ESpp4doUVkkON9lOINRflPxASauZAQ=; b=ESLaDSzuwyqyGRNA0wnezhtqgg5QoIY4maqdkKVNAB1//eIwIej0Tch4J/Ca7eYEKf a1QtGRIh0UML47+9RpJsIQviYHs/PwZ3oKXJ0CRt5FZyoLGmXKA5OuAOeVfYyKW5iExB 0h9M6g4emTSO2d/m8MT/rmRsDTPgSPfLxgjCN9PshDLdoIhu4Avk20DL5337Q1GnJOBf 9nafvPkdyFH6VV2GuS0MR01ooZ+UWUdFgO1KMHxALH1UKoVO/JaRR2PVIRf9FS3bfQU0 2TaDvv4ijvO3dBkwdX+cx5vGUn9+tksv3BOfZ4zzNtBux9xaMCk13m3IH32RTHQub4aI FQug== X-Gm-Message-State: AOAM530yes/vIgBOEe2AHL6954sQ9fTSbB4NoWJe3f9wJ1jD5p9000uB +weG4cKplWfkjYjLFGLxYu3cuRoX8OFkuMoxEPFirkfBgHEL0zfDHDPN2nYcBDaKBuTvJn2FXYD U35U+ib/PwdEdsAMVafK4EXmEIKC3u9kxOV2z/6P4nMjD4XeYzI9Jfz4f4Fni4MCSTTrE9Zo= X-Received: by 2002:a05:620a:46a3:b0:69f:47be:60b with SMTP id bq35-20020a05620a46a300b0069f47be060bmr11766587qkb.288.1651107310567; Wed, 27 Apr 2022 17:55:10 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzHGEckNP73Ezkr2M+ZIYS30eIllc72MV8E7WFIZ8P4P2ZC2XrlG0Mbm9fDIIhrhRMmK2DtYA== X-Received: by 2002:a05:620a:46a3:b0:69f:47be:60b with SMTP id bq35-20020a05620a46a300b0069f47be060bmr11766573qkb.288.1651107310054; Wed, 27 Apr 2022 17:55:10 -0700 (PDT) Received: from smtpclient.apple ([47.208.199.57]) by smtp.gmail.com with ESMTPSA id a23-20020a05620a103700b00699d49c511dsm8660587qkk.104.2022.04.27.17.55.09 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Apr 2022 17:55:09 -0700 (PDT) From: Ben Woodard Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.80.82.1.1\)) Subject: Fwd: Testing Setup - More Tests and Automation? Date: Wed, 27 Apr 2022 17:55:07 -0700 References: To: Ben Woodard via Libabigail Message-Id: <34E58964-E930-4DF9-87CD-18D4C63DBCEB@redhat.com> X-Mailer: Apple Mail (2.3696.80.82.1.1) X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-5.5 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2022 00:55:14 -0000 My coworker who did most of the work included in: https://sourceware.org/pi= permail/libabigail/2022q2/004292.html asked me to forward this email along to the mailing list be= cause her posts are getting marked as spam. I personally feel like we need to reconsider https://sourceware.org/piperma= il/libabigail/2022q1/004045.html and have checks like this built into the n= ormal =E2=80=9Cmake check=E2=80=9D having ENABLE_SLOW_TEST=3Dno by default = allows too many bugs to creep in. People don=E2=80=99t use it. As a case in= point neither the current trunk or the fixes branch pass all the tests whe= n it is set.=20 I will say Vanessa has convinced me of the value of the automated testing. = It is great when she does a pull request, I can immediately see not only th= at it applies but that it continues to pass all the regression tests. I fee= l like it would be helpful to have more tests included within the make chec= k so that it didn=E2=80=99t depend on me to make sure that something isn=E2= =80=99t going wrong. -ben > Begin forwarded message: >=20 > From: v > Subject: Fwd: Testing Setup - More Tests and Automation? > Date: April 27, 2022 at 4:42:09 PM PDT > To: woodard@redhat.com >=20 > See my email below - I joined the sourceware list and it blocked me as sp= am. =F0=9F=98=AD >=20 > ---------- Forwarded message --------- > From: v > Date: Wed, Apr 27, 2022 at 5:40 PM > Subject: Testing Setup - More Tests and Automation? > To: >=20 >=20 > Hey Dodji et al, >=20 > I am writing in response to https://sourceware.org/pipermail/libabigail/2= 022q1/004045.html. I understand the desire to not add too many tests to mak= e check, but I think maybe there is a nice middle ground or compromise? >=20 > Basically, what I think libabigail needs is more automated testing of mor= e cases. I added a simple setup to Ben's repo that does an automated build = in a container, and since it starts with a base container with the deps it'= s not terribly too slow: >=20 > https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus= =3Dtrue >=20 > In that run we do make check (Ben had added extra tests there) but it wou= ld also be feasible to clone a testing repository instead. This is what I s= et up for dyninst, because their test suite is a bit of a chonker too (note= the red clone into external tests).=20 >=20 > https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=3Dtr= ue#step:5:78 >=20 > Also, it would be pretty cool to run libabigail on itself to check for br= eaks: >=20 > https://github.com/woodard/libabigail/actions/runs/2210184567 >=20 > And (kind of cool) to always run the changes in the PR against a bunch of= known distro libs (here for Fedora). >=20 > https://github.com/woodard/libabigail/actions/runs/2208891579 >=20 > So, I think what I'm saying is that adding automation to testing libabiga= il would really empower us to catch more bugs. Have y'all ever considered m= oving some project stuffs over to GitHub so more people can help with libai= bgail than just you and Ben? And we can implement a lot of automation for t= he project proper? I gave a talk recently at CU Boulder, and the students w= ere really interested in libabigail. If it's readily available to contribut= e to on GitHub, I (and others I bet) would really enjoy contributing. I alm= ost didn't write this email because it's a PITA to have to join a mailing l= ist, but I think it's important. If you forever keep it on sourceware (chan= neling 1993 they want their web design back!) I think in the long run it's = more work for you, and probably less fun too. >=20 > What do you think? Is this a future you could imagine? And how might we g= et there? >=20 > Best, >=20 > Vanessa