From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 5381 invoked by alias); 2 Aug 2007 08:12:00 -0000 Received: (qmail 5367 invoked by uid 22791); 2 Aug 2007 08:12:00 -0000 X-Spam-Status: No, hits=-2.1 required=5.0 tests=AWL,BAYES_00,DK_POLICY_SIGNSOME,FORGED_RCVD_HELO X-Spam-Check-By: sourceware.org Received: from wildebeest.demon.nl (HELO gnu.wildebeest.org) (83.160.170.119) by sourceware.org (qpsmtpd/0.31) with ESMTP; Thu, 02 Aug 2007 08:11:57 +0000 Received: from dijkstra.wildebeest.org ([192.168.1.29]) by gnu.wildebeest.org with esmtp (Exim 4.43) id 1IGVpM-0006mc-PN; Thu, 02 Aug 2007 10:14:37 +0200 Subject: Re: Testing frysk.testbed.Funit* From: Mark Wielaard To: Andrew Cagney Cc: frysk@sourceware.org In-Reply-To: <46AF5BDE.6000806@redhat.com> References: <46AE081A.8070503@redhat.com> <1185876171.3653.64.camel@dijkstra.wildebeest.org> <46AF5BDE.6000806@redhat.com> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-6H2nhyZ1PSPQSufH9suY" Date: Thu, 02 Aug 2007 08:12:00 -0000 Message-Id: <1186042314.15044.35.camel@dijkstra.wildebeest.org> Mime-Version: 1.0 X-Mailer: Evolution 2.8.3 (2.8.3-2.fc6) X-Spam-Score: -4.4 (----) X-Virus-Checked: Checked by ClamAV on sourceware.org X-IsSubscribed: yes Mailing-List: contact frysk-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: frysk-owner@sourceware.org X-SW-Source: 2007-q3/txt/msg00258.txt.bz2 --=-6H2nhyZ1PSPQSufH9suY Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Content-length: 2776 Hi Andrew, On Tue, 2007-07-31 at 11:57 -0400, Andrew Cagney wrote: > Mark Wielaard wrote: > > On Mon, 2007-07-30 at 11:47 -0400, Andrew Cagney wrote: > >=20=20=20 > >> I'm looking at ways to more directly test the frysk.testbed.Funit*=20 > >> classes (e.g., FunitExec, DetachedAckProcess) that wrap=20 > >> PKGLIBDIR/funit-* utilities, but am finding that the most effective=20 > >> route is to use frysk.proc's framework - duplicating the existing=20 > >> frysk.proc tests exercising frysk.proc functionality that is effective= ly=20 > >> testing that code. > >> > >> I could duplicate the tests but it seems redundant. Any thoughts on a= =20 > >> strategy? > > > > I might be missing the exact cases you want to test. But can't you just > > audit the current frysk.proc tests to see if they cover all relevant > > cases already and if not add one or two tests to the existing proc tests > > so all cases are covered? That way you will also extend the real proc > > tests to handle more cases catching two birds with one stone (if that > > isn't a terribly political incorrect saying). > > > you describe the current state of play; frysk.proc code is testing both=20 > itself internally and the funit tools implicitly. There's nothing=20 > directly testing units such as FunitExecOffspring; instead it is done=20 > implicitly via frysk.proc. That is great when it works, but not so=20 > great when tests fail as differentiating between an FunitExecOffspring=20 > or frysk.proc breakage that caused the fail isn't possible. Tests that not just cover the bare essentials, but test the code in actual use scenarios are very important. It makes sure that the code is tested as it will actually be used. And in this case, if you find something not covered, the proc code gets also more tests. As you say that is great if it works. But I get the feeling that is not enough for your current strategy. > As a contrasting example. Say we find the UI is crashing and track it=20 > down to a dwarf binding bug. What we do is add a test-case to the dwarf= =20 > bindings testing the problem (which contains the root cause of the=20 > problem); and then fix it. Is the core code is now being tested, we're=20 > confident that our problem won't return. Is there an effective way to=20 > do that here with the Funit* bindings? Right. That is what I am actually suggesting. Make sure that there are enough tests in proc that you feel confident that everything under funit is covered. Then if some issue is found anyway later on and it is tracked down to funit then just add an extra test there when you fix the problem so you can be confident it won't return. Besides that you have to fall back on your suggestion, slightly redundant, test duplication I am afraid. Cheers, Mark --=-6H2nhyZ1PSPQSufH9suY Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part Content-length: 189 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) iD8DBQBGsZHCxVhZCJWr9QwRAu61AJkByjJSBSKm9peaOPQ72hxiCYMZgwCfQHjE 5ZbWubcLaHW9NFW85bWIQF8= =iWlD -----END PGP SIGNATURE----- --=-6H2nhyZ1PSPQSufH9suY--