From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by sourceware.org (Postfix) with ESMTPS id C8B593848429; Fri, 16 Jul 2021 17:26:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C8B593848429 Received: by mail-qt1-x829.google.com with SMTP id v14so7623367qtc.8; Fri, 16 Jul 2021 10:26:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=GQKM8izmy0z3l+k6XgF07TSu/XioQA8lH3/HrCuHJq8=; b=oYk2NpkQli5Wf3/2oQL6mx2adSIWfF7pgN+IWvgByt+T0YYGsaqqMFXSMsTGYo6hHW jg9AvdlltAXV9k6QSHWUJYESzamC9Q56TtzyPHAVS0NQfNZ4vbEfQ5XLMjmDP6R0W4pL kG0L4fTEAVWJI8sUmF9zZxJOaHF2cpeOieWKCE6ZXc4bO1TB+45y/bX89q3ev1A5V0JX 0ThOGpVFHv8ljxPhTCg/MbDeZH9QsqRFveEUtWxoCXnTscKoyyyw5py7+dVcpJ11XGtR YCTXmax/qfSIGrk1SdSugwjksDS/rZrsEZBvOQEtIIMWv6MFvNXk95oR+Z4DSdRNqrGN NcnA== X-Gm-Message-State: AOAM530NeG19Y/RQhR+ZLHqwuAPuwRv1jlEaciybhGRADas/pgDQ0AZf 2C3lg0gXpG/I2FtukNSHXLGtZdJDoyI= X-Google-Smtp-Source: ABdhPJzb82wcxtFU1EoYorSLJfqDUPe3qPoodSIgSmF6R18282fm97LC8uGgCQntyUmmik3xUoESzg== X-Received: by 2002:ac8:7953:: with SMTP id r19mr9988551qtt.28.1626456405303; Fri, 16 Jul 2021 10:26:45 -0700 (PDT) Received: from [192.168.0.41] (75-166-102-22.hlrn.qwest.net. [75.166.102.22]) by smtp.gmail.com with ESMTPSA id k9sm65601qti.88.2021.07.16.10.26.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 16 Jul 2021 10:26:44 -0700 (PDT) Subject: Re: Pushing XFAILed test cases To: Thomas Schwinge , Thomas Koenig , gcc@gcc.gnu.org Cc: Sandra Loosemore , gcc-patches@gcc.gnu.org, fortran@gcc.gnu.org References: <13168f92-8863-cb63-9470-a6055d5da5f6@codesourcery.com> <10658f98-0a72-e80d-0cc6-7b4624eea1f1@netcologne.de> <87im1ab7g4.fsf@euler.schwinge.homeip.net> From: Martin Sebor Message-ID: <89ed1e8e-3f99-4c1e-7def-86b3bdf62e6f@gmail.com> Date: Fri, 16 Jul 2021 11:26:43 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: <87im1ab7g4.fsf@euler.schwinge.homeip.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, KAM_SHORT, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jul 2021 17:26:47 -0000 On 7/16/21 9:32 AM, Thomas Schwinge wrote: > [Also including for guidance.] > > > Hi! > > (I'm not involved in or familiar with Sandra's Fortran TS29113 work, just > commenting generally here.) > > > On 2021-07-16T09:52:28+0200, Thomas Koenig via Gcc-patches wrote: >> It is my understanding that it is not gcc policy to add xfailed test >> cases for things that do not yet work. Rather, xfail is for tests that >> later turn out not to work, especially on certain architectures. > > That's not current practice, as far as I can tell. I'm certainly > "guilty" of pushing lots of XFAILed test cases (or, most often, > individual XFAILed DejaGnu directives), and I see a good number of others > GCC folks do that, too. Ideally with but casually also without > corresponding GCC PRs filed. If without, then of course should have > suitable commentary inside the test case file. Time span of addressing > the XFAILs ranging between days and years. > > In my opinion, if a test case has been written and analyzed, why > shouldn't you push it, even if (parts of) it don't quite work yet? (If > someone -- at another time, possibly -- then implements the missing > functionality/fixes the bugs, the XFAILs turn into XPASSes, thus serving > to demonstrate the effect of code changes. > > Otherwise -- and I've run into that just yesterday... -- effort spent on > such test cases simply gets lost "in the noise of the mailing list > archives", until re-discovered, or -- in my case -- re-implemented and > then re-discovered by chance. > > We nowadays even have a way to mark up ICEing test cases ('dg-ice'), > which has been used to push test cases that ICE for '{ target *-*-* }'. > > > Of course, we shall assume a certain level of quality in the XFAILed test > cases: I'm certainly not suggesting we put any random junk into the > testsuite, coarsely XFAILed. (I have not reviewed Sandra's test cases to > that effect, but knowing here, I'd be surprised if that were the problem > here.) > > > Not trying to overrule you, just sharing my opinion -- now happy to hear > others. :-) I've also been xfailing individual directives in new tests, with or without PRs tracking the corresponding limitations (not so much outright bugs as future enhancements). The practice has been discussed in the past and (IIRC) there was general agreement with it. Marek even formalized some of it for the C++ front end by adding support for one or more dg- directives (I think dg-ice was one of them). The discussion I recall is here: https://gcc.gnu.org/pipermail/gcc-patches/2020-July/550913.html Martin