From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) by sourceware.org (Postfix) with ESMTPS id BB2523858D1E for ; Thu, 28 Apr 2022 01:57:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org BB2523858D1E Received: by mail-ej1-x634.google.com with SMTP id bv19so6725448ejb.6 for ; Wed, 27 Apr 2022 18:57:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=98kBy7xCbPEIrVulk8kQkp89U37MFoiwCMdugagkv9E=; b=JKj5cwp7G2IFK4Ci3cnOrKt+vAayl05IM62ZX8AO1qMPz/W39qEcdCjrGc5xdXxMVz z49zMf2yunip8Z5Ok2z/mRhkMLw2KZ1cr0L7dCkfunDvmSFykuoZAxynIyHH9NwruqTQ zHTdiX0sH74DT7mhllcTTj/rTkvyvVHply2dVCa+GY4OWtRNKPgN78whqVg6jHdMYZXN CPBUraZqsgj1dKeEQlYbmlADkk8Y4rcgnH7DxSvrzE/c7SeVj/CSOFNs5zxkwTRF2bgp aCB/KcayyTg10HzfUQE2NZHL7ELrUmVTRBO0cSkoSxtIX+2huIclAlO4RflKrjWCDGB0 pGXg== X-Gm-Message-State: AOAM531JD2qvioKHQRWznDydPpaxaVelTUvE+tIvbgzQuHfqxL785u8I 4QDknjgHd65TIs9Kr7dJ3LsuPKMFut80SmXg/M+u1tgS X-Google-Smtp-Source: ABdhPJxSdwK+ZqtrcNseo93yTzBC6lsG83qSEIWSVLtmTYBPN/nuwiOI5hsq6ta87fPDQAcjTzkEtdWuUzYo3Dd8BXo= X-Received: by 2002:a17:907:7ea9:b0:6f3:de9c:c6fb with SMTP id qb41-20020a1709077ea900b006f3de9cc6fbmr932866ejc.304.1651111058885; Wed, 27 Apr 2022 18:57:38 -0700 (PDT) MIME-Version: 1.0 References: <34E58964-E930-4DF9-87CD-18D4C63DBCEB@redhat.com> In-Reply-To: <34E58964-E930-4DF9-87CD-18D4C63DBCEB@redhat.com> From: v Date: Wed, 27 Apr 2022 19:57:12 -0600 Message-ID: Subject: Re: Testing Setup - More Tests and Automation? To: Ben Woodard Cc: Ben Woodard via Libabigail X-Spam-Status: No, score=2.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, HTML_MESSAGE, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Apr 2022 01:57:43 -0000 Going to give this one more shot in case it works without the image (and I don't get flagged as spam...) if you didn't see my attached message below Ben's, please unwrap the email and look! =F0=9F=91=87 Also Dodji I heard from a little bird that you are competing in a national championship this weekend? Kick @ss!! Just forget about work until then because being a national champion (or even contender) is so much cooler :) On Wed, Apr 27, 2022 at 6:55 PM Ben Woodard wrote: > My coworker who did most of the work included in: > https://sourceware.org/pipermail/libabigail/2022q2/004292.html sadly got scrubbed due to a bug in Fedora 36> asked me to forward this > email along to the mailing list because her posts are getting marked as > spam. > > I personally feel like we need to reconsider > https://sourceware.org/pipermail/libabigail/2022q1/004045.html and have > checks like this built into the normal =E2=80=9Cmake check=E2=80=9D havin= g > ENABLE_SLOW_TEST=3Dno by default allows too many bugs to creep in. People > don=E2=80=99t use it. As a case in point neither the current trunk or the= fixes > branch pass all the tests when it is set. > > I will say Vanessa has convinced me of the value of the automated testing= . > It is great when she does a pull request, I can immediately see not only > that it applies but that it continues to pass all the regression tests. I > feel like it would be helpful to have more tests included within the make > check so that it didn=E2=80=99t depend on me to make sure that something = isn=E2=80=99t > going wrong. > > -ben > > > > Begin forwarded message: > > > > From: v > > Subject: Fwd: Testing Setup - More Tests and Automation? > > Date: April 27, 2022 at 4:42:09 PM PDT > > To: woodard@redhat.com > > > > See my email below - I joined the sourceware list and it blocked me as > spam. =F0=9F=98=AD > > > > ---------- Forwarded message --------- > > From: v > > Date: Wed, Apr 27, 2022 at 5:40 PM > > Subject: Testing Setup - More Tests and Automation? > > To: > > > > > > Hey Dodji et al, > > > > I am writing in response to > https://sourceware.org/pipermail/libabigail/2022q1/004045.html. I > understand the desire to not add too many tests to make check, but I thin= k > maybe there is a nice middle ground or compromise? > > > > Basically, what I think libabigail needs is more automated testing of > more cases. I added a simple setup to Ben's repo that does an automated > build in a container, and since it starts with a base container with the > deps it's not terribly too slow: > > > > > https://github.com/woodard/libabigail/runs/6133179077?check_suite_focus= =3Dtrue > > > > In that run we do make check (Ben had added extra tests there) but it > would also be feasible to clone a testing repository instead. This is wha= t > I set up for dyninst, because their test suite is a bit of a chonker too > (note the red clone into external tests). > > > > > https://github.com/dyninst/dyninst/runs/6187357967?check_suite_focus=3Dtr= ue#step:5:78 > > > > Also, it would be pretty cool to run libabigail on itself to check for > breaks: > > > > https://github.com/woodard/libabigail/actions/runs/2210184567 > > > > And (kind of cool) to always run the changes in the PR against a bunch > of known distro libs (here for Fedora). > > > > https://github.com/woodard/libabigail/actions/runs/2208891579 > > > > So, I think what I'm saying is that adding automation to testing > libabigail would really empower us to catch more bugs. Have y'all ever > considered moving some project stuffs over to GitHub so more people can > help with libaibgail than just you and Ben? And we can implement a lot of > automation for the project proper? I gave a talk recently at CU Boulder, > and the students were really interested in libabigail. If it's readily > available to contribute to on GitHub, I (and others I bet) would really > enjoy contributing. I almost didn't write this email because it's a PITA = to > have to join a mailing list, but I think it's important. If you forever > keep it on sourceware (channeling 1993 they want their web design back!) = I > think in the long run it's more work for you, and probably less fun too. > > > > What do you think? Is this a future you could imagine? And how might we > get there? > > > > Best, > > > > Vanessa > >