From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTPS id 1741F3858D33 for ; Thu, 16 Feb 2023 09:38:12 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1741F3858D33 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676540291; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references; bh=ARHRqoycO8Y6eVtVbwNc5EBB93mCLaYskOaHheks4bU=; b=BiVT1qY/8z5ACAUVfzkwJOGJd8k+kGiBdNQyN7YVpIrczPTLQOlLnxfgF6+vDKJo+AThNq oS4DP/sz66LTHvX8/p+gzP1b2WynwCteRYeiF9V4wRR6V/SP76SiItYJhD4DjSiwMJS9Rf yeMRZHiWOTmL6cRaFGr6/KoLCkpTV1k= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-249-4eYhOLFOOv6Dv7GmmuPPAw-1; Thu, 16 Feb 2023 04:38:08 -0500 X-MC-Unique: 4eYhOLFOOv6Dv7GmmuPPAw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 943221C189B1; Thu, 16 Feb 2023 09:38:07 +0000 (UTC) Received: from tucnak.zalov.cz (unknown [10.39.193.203]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 37ADEC15BA0; Thu, 16 Feb 2023 09:38:07 +0000 (UTC) Received: from tucnak.zalov.cz (localhost [127.0.0.1]) by tucnak.zalov.cz (8.17.1/8.17.1) with ESMTPS id 31G9c4Np2901979 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Thu, 16 Feb 2023 10:38:04 +0100 Received: (from jakub@localhost) by tucnak.zalov.cz (8.17.1/8.17.1/Submit) id 31G9c3aJ2901978; Thu, 16 Feb 2023 10:38:03 +0100 Date: Thu, 16 Feb 2023 10:38:03 +0100 From: Jakub Jelinek To: juzhe.zhong@rivai.ai Cc: gcc-patches@gcc.gnu.org, kito.cheng@gmail.com, Jeff Law Subject: Re: [PATCH] RISC-V: Add vm* mask C api tests Message-ID: Reply-To: Jakub Jelinek References: <20230216033619.16472-1-juzhe.zhong@rivai.ai> MIME-Version: 1.0 In-Reply-To: <20230216033619.16472-1-juzhe.zhong@rivai.ai> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Status: No, score=-3.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,KAM_SHORT,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi! I see in the past few weeks you've added huge amounts of these tests du -shc *.target/riscv/*/ 34M gcc.target/riscv/rvv/ 28M g++.target/riscv/rvv/ 61M total and new are coming (nothing at all at this year's start). This is far larger than tests of any other architecture (i386 has 35M total, aarch64 31M total, arm 17M total, powerpc 12M total, everything else is even much smaller) but for the other architectures it has been decades of testsuite coverage for features added over the years. Rather than looking purely at size, I'm more worried about the content of the tests. Usually target testsuites include runtime tests whether particular intrinsics etc. behave correctly at runtime, plus some compile tests that they can be compiled with occassional scan-assembler* to mention a particular instruction appears, but in these cases the scan-assembler* covers the entire (albeit small) functions, which makes it IMHO a maintainance nightmare whenever one wants to change something important in the compiler. Take e.g. the recent Andreas Schwab's change to make -fasynchronous-unwind-tables the default on riscv, even that change required quite a few changes. My worry is that with these kind of tests changes like that will become much harder and some people will simply decide not to do such changes because having to adjust tens of thousands of tests even with some scripting would be a nightmare. Can't we do better than this? E.g. what is the difference between gcc.target/riscv/rvv/ and g++.target/riscv/rvv/ tests? Are the APIs so different between C and C++ that it needs to be tested twice? Even if so, we have the concept of c-c++-common tests, we could add c-c++-common.target and make riscv.exp handle it similarly to how e.g. C and C++ dg.exp handles those. How do you create these tests? If you use some generator for them, wouldn't it be better to include the generator in the testsuite and generate them on the fly? We already have a precedent for that, e.g. the gcc/testsuite/g*.dg/compat/struct-layout-1.exp testsuite has a generator program written in C that creates tests on the fly. Now, using something like that would have 2 advantages, it would be much easier for maintainance, if you do some global change in the compiler that affects those tests, just adjust a few spots in the generator instead of tweaking currently 6000 tests and counting. Even if you aren't using a generator to write these tests (that would be a lot of work then!), a question is if it couldn't be done by one, have say some file like gcc has *.def files all around to describe what you want to test and something that generates those. Just wanted to chime in before we have 10 times more of such tests and it will be too late to adjust... On Thu, Feb 16, 2023 at 11:36:19AM +0800, juzhe.zhong@rivai.ai wrote: > From: Ju-Zhe Zhong > > gcc/testsuite/ChangeLog: > > * gcc.target/riscv/rvv/base/vmand_mm-1.c: New test. > * gcc.target/riscv/rvv/base/vmand_mm-2.c: New test. > * gcc.target/riscv/rvv/base/vmand_mm-3.c: New test. > * gcc.target/riscv/rvv/base/vmandn_mm-1.c: New test. > * gcc.target/riscv/rvv/base/vmandn_mm-2.c: New test. > * gcc.target/riscv/rvv/base/vmandn_mm-3.c: New test. > * gcc.target/riscv/rvv/base/vmclr_m_m-1.c: New test. > * gcc.target/riscv/rvv/base/vmclr_m_m-2.c: New test. > * gcc.target/riscv/rvv/base/vmclr_m_m-3.c: New test. > * gcc.target/riscv/rvv/base/vmmv_m_m-1.c: New test. > * gcc.target/riscv/rvv/base/vmmv_m_m-2.c: New test. > * gcc.target/riscv/rvv/base/vmmv_m_m-3.c: New test. > * gcc.target/riscv/rvv/base/vmnand_mm-1.c: New test. > * gcc.target/riscv/rvv/base/vmnand_mm-2.c: New test. > * gcc.target/riscv/rvv/base/vmnand_mm-3.c: New test. > * gcc.target/riscv/rvv/base/vmnor_mm-1.c: New test. > * gcc.target/riscv/rvv/base/vmnor_mm-2.c: New test. > * gcc.target/riscv/rvv/base/vmnor_mm-3.c: New test. > * gcc.target/riscv/rvv/base/vmnot_m_m-1.c: New test. > * gcc.target/riscv/rvv/base/vmnot_m_m-2.c: New test. > * gcc.target/riscv/rvv/base/vmnot_m_m-3.c: New test. > * gcc.target/riscv/rvv/base/vmor_mm-1.c: New test. > * gcc.target/riscv/rvv/base/vmor_mm-2.c: New test. > * gcc.target/riscv/rvv/base/vmor_mm-3.c: New test. > * gcc.target/riscv/rvv/base/vmorn_mm-1.c: New test. > * gcc.target/riscv/rvv/base/vmorn_mm-2.c: New test. > * gcc.target/riscv/rvv/base/vmorn_mm-3.c: New test. > * gcc.target/riscv/rvv/base/vmsbf_m_m-1.c: New test. > * gcc.target/riscv/rvv/base/vmsbf_m_m-2.c: New test. > * gcc.target/riscv/rvv/base/vmsbf_m_m-3.c: New test. > * gcc.target/riscv/rvv/base/vmsbf_m_mu-1.c: New test. > * gcc.target/riscv/rvv/base/vmsbf_m_mu-2.c: New test. > * gcc.target/riscv/rvv/base/vmsbf_m_mu-3.c: New test. > * gcc.target/riscv/rvv/base/vmset_m_m-1.c: New test. > * gcc.target/riscv/rvv/base/vmset_m_m-2.c: New test. > * gcc.target/riscv/rvv/base/vmset_m_m-3.c: New test. > * gcc.target/riscv/rvv/base/vmsif_m_m-1.c: New test. > * gcc.target/riscv/rvv/base/vmsif_m_m-2.c: New test. > * gcc.target/riscv/rvv/base/vmsif_m_m-3.c: New test. > * gcc.target/riscv/rvv/base/vmsif_m_mu-1.c: New test. > * gcc.target/riscv/rvv/base/vmsif_m_mu-2.c: New test. > * gcc.target/riscv/rvv/base/vmsif_m_mu-3.c: New test. > * gcc.target/riscv/rvv/base/vmsof_m_m-1.c: New test. > * gcc.target/riscv/rvv/base/vmsof_m_m-2.c: New test. > * gcc.target/riscv/rvv/base/vmsof_m_m-3.c: New test. > * gcc.target/riscv/rvv/base/vmsof_m_mu-1.c: New test. > * gcc.target/riscv/rvv/base/vmsof_m_mu-2.c: New test. > * gcc.target/riscv/rvv/base/vmsof_m_mu-3.c: New test. > * gcc.target/riscv/rvv/base/vmxnor_mm-1.c: New test. > * gcc.target/riscv/rvv/base/vmxnor_mm-2.c: New test. > * gcc.target/riscv/rvv/base/vmxnor_mm-3.c: New test. > * gcc.target/riscv/rvv/base/vmxor_mm-1.c: New test. > * gcc.target/riscv/rvv/base/vmxor_mm-2.c: New test. > * gcc.target/riscv/rvv/base/vmxor_mm-3.c: New test. Jakub