From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id D0C6638582B5 for ; Mon, 12 Sep 2022 07:52:01 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org D0C6638582B5 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-661-BU6Q7jshN8-CzRyE_kiMPA-1; Mon, 12 Sep 2022 03:52:00 -0400 X-MC-Unique: BU6Q7jshN8-CzRyE_kiMPA-1 Received: by mail-wm1-f71.google.com with SMTP id r186-20020a1c44c3000000b003b3309435a9so6303167wma.6 for ; Mon, 12 Sep 2022 00:51:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date; bh=4PrBGwTdwKYvNEi4wvow31O5kFWk9bmj63cZIViyFek=; b=JksmcAyRn6f7vxA1zO0tuL+EnWtINXCRLS12bCV9HclMn+pgzE9gnUbCKoFFVSbn8B QvrP6CVpf7TDa57aPtZ8BSVOPkwSL6DSho3ZYMY3NMkcVboDf4ldy68No1/UgvNhadjq /GRXJ96T+ewwvb93Z+ocqhX3QNVBrcooQXOi9SGCHbVh5kJjaEI6aThgvpM94qTNJ0px Y2WZHNQ1256L9htC6rxqzQFPxdhir0mDJwdcDQnVQhAEa7qihzHcb5YRFNVUX6zWTDTU t0LphlQRqt0hcBay8Eo55fHA91DBk8RSn8mpjJEEuNahHvmP42yPWPoZ6KThiX+jvyj0 L1tg== X-Gm-Message-State: ACgBeo0oopx5P8lPRCrKSJsFfEhEAqny6+SV9YQr8SRQ2dq9SPvd/vS/ PTefMenqLJhctt6kBcVHwcTMW6soN9DevNWDFruao2zpE+TgPUKaovVbIq+dG7T5JI7KOVsWEax kXbO8n/5zUkSR38AtJ0jb2Q== X-Received: by 2002:a05:600c:1d94:b0:3b4:7b91:7056 with SMTP id p20-20020a05600c1d9400b003b47b917056mr3899130wms.18.1662969118528; Mon, 12 Sep 2022 00:51:58 -0700 (PDT) X-Google-Smtp-Source: AA6agR75pxI29s3m6w6c9wHvrUQzmHaC/5BPSfbE6o8Dn6aUcb+cR3MJ2HMfe9J6ChpLsoxLTNh2uw== X-Received: by 2002:a05:600c:1d94:b0:3b4:7b91:7056 with SMTP id p20-20020a05600c1d9400b003b47b917056mr3899116wms.18.1662969118224; Mon, 12 Sep 2022 00:51:58 -0700 (PDT) Received: from [10.43.2.105] (nat-pool-brq-t.redhat.com. [213.175.37.10]) by smtp.gmail.com with ESMTPSA id f12-20020a05600c154c00b003a5f3f5883dsm9651796wmg.17.2022.09.12.00.51.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 12 Sep 2022 00:51:57 -0700 (PDT) Message-ID: Date: Mon, 12 Sep 2022 09:51:57 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.1 Subject: Re: [PATCH] gdb/testsuite: handle invalid .exp names passed in TESTS To: Andrew Burgess , gdb-patches@sourceware.org References: <20220910164349.1945521-1-aburgess@redhat.com> From: Bruno Larsen In-Reply-To: <20220910164349.1945521-1-aburgess@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-13.9 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, NICE_REPLY_A, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Sep 2022 07:52:03 -0000 Hi Andrew, Once again, very well timed patch as I had just noticed this problem! Other than a typo (response inlined), my only comment is that, since you are already renaming check-single, what do you think about renaming to check-single-thread or check-sequential? My reasoning is that when first reading the message and diff, I thought this rule would test a single test, which left me confused for a minute. Either way, I like the patch, and encourage you to approve it! -- Cheers, Bruno On 10/09/2022 18:43, Andrew Burgess via Gdb-patches wrote: > I ran some tests like: > > $ make check-gdb TESTS="gdb.base/break.exp" > > then, then I went to rerun the tests later, I managed to corrupt the > command line, like this: > > $ make check-gdb TESTS="gdb.base/breakff.exp" > > the make command did exit with an error, but DejaGnu appeared to > report that every test passed! The tail end of the output looks like > this: > > Illegal Argument "no-matching-tests-found" > try "runtest --help" for option list > === gdb Summary === > > # of expected passes 115 > /tmp/build/gdb/gdb version 13.0.50.20220831-git -nw -nx -iex "set height 0" -iex "set width 0" -data-directory /tmp/build/gdb/testsuite/../data-directory > > make[3]: *** [Makefile:212: check-single] Error 1 > make[3]: Leaving directory '/tmp/build/gdb/testsuite' > make[2]: *** [Makefile:161: check] Error 2 > make[2]: Leaving directory '/tmp/build/gdb/testsuite' > make[1]: *** [Makefile:1916: check] Error 2 > make[1]: Leaving directory '/tmp/build/gdb' > make: *** [Makefile:13565: check-gdb] Error 2 > > For a while, I didn't spot that DejaGnu had failed at all, I saw the > 115 passes, and thought everything had run correctly - though I was > puzzled that make was reporting an error. > > What happens is that in gdb/testsuite/Makefile, in the check-single > rule, we first run DejaGnu, then run the dg-add-core-file-count.sh > script, and finally, we use sed to extract the results from the > gdb.sum file. > > In my case, with the invalid test name, DejaGnu fails, but the > following steps are still run, the final result, the 115 passes, is > then extracted from the pre-existing gdb.sum file. > > If I use 'make -jN' then the 'check-parallel' rule, rather than the > 'check-single' rule is used. In this case the behaviour is slightly > different, the tail end of the output now looks like this: > > No matching tests found. > > make[4]: Leaving directory '/tmp/build/gdb/testsuite' > find: ‘outputs’: No such file or directory > Usage: ../../../src/gdb/testsuite/../../contrib/dg-extract-results.py [-t tool] [-l variant-list] [-L] log-or-sum-file ... > > tool The tool (e.g. g++, libffi) for which to create a > new test summary file. If not specified then output > is created for all tools. > variant-list One or more test variant names. If the list is > not specified then one is constructed from all > variants in the files for . > sum-file A test summary file with the format of those > created by runtest from DejaGnu. > If -L is used, merge *.log files instead of *.sum. In this > mode the exact order of lines may not be preserved, just different > Running *.exp chunks should be in correct order. > find: ‘outputs’: No such file or directory > Usage: ../../../src/gdb/testsuite/../../contrib/dg-extract-results.py [-t tool] [-l variant-list] [-L] log-or-sum-file ... > > tool The tool (e.g. g++, libffi) for which to create a > new test summary file. If not specified then output > is created for all tools. > variant-list One or more test variant names. If the list is > not specified then one is constructed from all > variants in the files for . > sum-file A test summary file with the format of those > created by runtest from DejaGnu. > If -L is used, merge *.log files instead of *.sum. In this > mode the exact order of lines may not be preserved, just different > Running *.exp chunks should be in correct order. > make[3]: Leaving directory '/tmp/build/gdb/testsuite' > make[2]: Leaving directory '/tmp/build/gdb/testsuite' > make[1]: Leaving directory '/tmp/build/gdb' > > Rather than DejaGnu failing, we now get a nice 'No matching tests > found' message, followed by some other noise. This other noise is > first `find` failing, followed by the dg-extract-results.py script > failing. > > What happens here is that, in the check-parallel rule, the outputs > directory is deleted before DejaGnu is invoked. Then we try to run > all the tests, and finally we use find and dg-extract-results.py to > combine all the separate .sun .sum* > and .log files together. However, if > there are no tests run then the outputs/ directory is never created, > so the find command and consequently the dg-extract-results.py script, > fail. > > This commit aims to fix the following issues: > > (1) For check-single and check-parallel rules, don't run any of the > post-processing steps if DejaGnu failed to run. This will avoid all > the noise after the initial failure of DejaGnu, > > (2) For check-single ensure that we don't accidentally report > previous results, this is related to the above, but is worth calling > out as a separate point, and > > (3) For check-single, print the 'No matching tests found' message > just like we do for a parallel test run. This makes the parallel and > non-parallel testing behaviour more similar, and I think is clearer > than the current 'Illegal Argument' error message. > > Points (1) and (2) will be handled by moving the post processing steps > inside an if block within the recipe. For check-single I propose > deleting the gdb.sum and gdb.log files before running DejaGnu, this is > similar (I think) to how we delete the outputs/ directory in the > check-parallel rule. > > For point (3) I plan to split the check-single rule in two, the > existing check-single will be renamed do-check-single, then a new > check-single rule will be added. The new check-single rule can either > depend on the new do-check-single, or will ensure the 'No matching > tests found' message is printed when appropriate. > --- > gdb/testsuite/Makefile.in | 40 +++++++++++++++++++++++++++++---------- > 1 file changed, 30 insertions(+), 10 deletions(-) > > diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in > index 87ba522c9e0..49b8a60a35c 100644 > --- a/gdb/testsuite/Makefile.in > +++ b/gdb/testsuite/Makefile.in > @@ -203,18 +203,36 @@ expanded_tests := $(patsubst $(srcdir)/%,%,$(wildcard $(addprefix $(srcdir)/,$(T > expanded_tests_or_none := $(or $(expanded_tests),no-matching-tests-found) > endif > > +# With check-single, if TESTS was expanded to "no-matching-tests-found" then > +# this will be passed to DejaGnu, resuling in an error. With check-parallel > +# in the same situation, we avoid invoking DejaGnu, and instead just call > +# the check/no-matching-tests-found rule (which prints a helpful message). > +# > +# To get the same behaviour for check-single we decide here, based on how > +# TESTS expanded, whether check-single should redirect to do-check-single or > +# to check/no-matching-tests-found. > +ifeq ($(expanded_tests_or_none),no-matching-tests-found) > +CHECK_SINGLE_DEP=check/no-matching-tests-found > +else > +CHECK_SINGLE_DEP=do-check-single > +endif > + > # Shorthand for running all the tests in a single directory. > check-gdb.%: > $(MAKE) check TESTS="gdb.$*/*.exp" > > -check-single: > - -rm -f *core* > +do-check-single: > + -rm -f *core* gdb.sum gdb.log > $(DO_RUNTEST) $(RUNTESTFLAGS) $(expanded_tests_or_none) $(TIMESTAMP); \ > result=$$?; \ > - $(SHELL) $(srcdir)/lib/dg-add-core-file-count.sh; \ > - sed -n '/=== gdb Summary ===/,$$ p' gdb.sum; \ > + if test -e gdb.sum; then \ > + $(SHELL) $(srcdir)/lib/dg-add-core-file-count.sh; \ > + sed -n '/=== gdb Summary ===/,$$ p' gdb.sum; \ > + fi; \ > exit $$result > > +check-single: $(CHECK_SINGLE_DEP) > + > check-single-racy: > -rm -rf cache racy_outputs temp > mkdir -p racy_outputs; \ > @@ -240,12 +258,14 @@ check-parallel: > -rm -rf cache outputs temp > $(MAKE) -k do-check-parallel; \ > result=$$?; \ > - $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh \ > - `find outputs -name gdb.sum -print` > gdb.sum; \ > - $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh -L \ > - `find outputs -name gdb.log -print` > gdb.log; \ > - $(SHELL) $(srcdir)/lib/dg-add-core-file-count.sh; \ > - sed -n '/=== gdb Summary ===/,$$ p' gdb.sum; \ > + if test -d outputs; then \ > + $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh \ > + `find outputs -name gdb.sum -print` > gdb.sum; \ > + $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh -L \ > + `find outputs -name gdb.log -print` > gdb.log; \ > + $(SHELL) $(srcdir)/lib/dg-add-core-file-count.sh; \ > + sed -n '/=== gdb Summary ===/,$$ p' gdb.sum; \ > + fi; \ > exit $$result > > check-parallel-racy: