Re: [PATCH] RISC-V: Add vm* mask C api tests

[Jakub Jelinek <jakub@redhat.com>]

Hi!

I see in the past few weeks you've added huge amounts of these tests
du -shc *.target/riscv/*/
34M	gcc.target/riscv/rvv/
28M	g++.target/riscv/rvv/
61M	total
and new are coming (nothing at all at this year's start).
This is far larger than tests of any other architecture
(i386 has 35M total, aarch64 31M total, arm 17M total, powerpc 12M total,
everything else is even much smaller) but for the other architectures it has
been decades of testsuite coverage for features added over the years.
Rather than looking purely at size, I'm more worried about the content
of the tests.  Usually target testsuites include runtime tests whether
particular intrinsics etc. behave correctly at runtime, plus some compile
tests that they can be compiled with occassional scan-assembler* to mention
a particular instruction appears, but in these cases the scan-assembler*
covers the entire (albeit small) functions, which makes it IMHO a
maintainance nightmare whenever one wants to change something important
in the compiler.  Take e.g. the recent Andreas Schwab's change to make
-fasynchronous-unwind-tables the default on riscv, even that change required
quite a few changes.  My worry is that with these kind of tests changes like
that will become much harder and some people will simply decide not to do
such changes because having to adjust tens of thousands of tests even with
some scripting would be a nightmare.  Can't we do better than this?

E.g. what is the difference between gcc.target/riscv/rvv/ and
g++.target/riscv/rvv/ tests?  Are the <riscv_vector.h> APIs so different
between C and C++ that it needs to be tested twice?  Even if so,
we have the concept of c-c++-common tests, we could add c-c++-common.target
and make riscv.exp handle it similarly to how e.g. C and C++ dg.exp handles
those.  How do you create these tests?  If you use some generator for them,
wouldn't it be better to include the generator in the testsuite and generate
them on the fly?  We already have a precedent for that, e.g. the
gcc/testsuite/g*.dg/compat/struct-layout-1.exp testsuite has a generator
program written in C that creates tests on the fly.  Now, using something
like that would have 2 advantages, it would be much easier for maintainance,
if you do some global change in the compiler that affects those tests, just
adjust a few spots in the generator instead of tweaking currently 6000 tests
and counting.  Even if you aren't using a generator to write these tests
(that would be a lot of work then!), a question is if it couldn't be done by
one, have say some file like gcc has *.def files all around to describe what
you want to test and something that generates those.

Just wanted to chime in before we have 10 times more of such tests and it
will be too late to adjust...

On Thu, Feb 16, 2023 at 11:36:19AM +0800, juzhe.zhong@rivai.ai wrote:
> From: Ju-Zhe Zhong <juzhe.zhong@rivai.ai>
> 
> gcc/testsuite/ChangeLog:
> 
>         * gcc.target/riscv/rvv/base/vmand_mm-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmand_mm-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmand_mm-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmandn_mm-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmandn_mm-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmandn_mm-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmclr_m_m-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmclr_m_m-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmclr_m_m-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmmv_m_m-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmmv_m_m-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmmv_m_m-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmnand_mm-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmnand_mm-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmnand_mm-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmnor_mm-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmnor_mm-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmnor_mm-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmnot_m_m-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmnot_m_m-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmnot_m_m-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmor_mm-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmor_mm-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmor_mm-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmorn_mm-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmorn_mm-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmorn_mm-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmsbf_m_m-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmsbf_m_m-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmsbf_m_m-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmsbf_m_mu-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmsbf_m_mu-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmsbf_m_mu-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmset_m_m-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmset_m_m-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmset_m_m-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmsif_m_m-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmsif_m_m-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmsif_m_m-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmsif_m_mu-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmsif_m_mu-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmsif_m_mu-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmsof_m_m-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmsof_m_m-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmsof_m_m-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmsof_m_mu-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmsof_m_mu-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmsof_m_mu-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmxnor_mm-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmxnor_mm-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmxnor_mm-3.c: New test.
>         * gcc.target/riscv/rvv/base/vmxor_mm-1.c: New test.
>         * gcc.target/riscv/rvv/base/vmxor_mm-2.c: New test.
>         * gcc.target/riscv/rvv/base/vmxor_mm-3.c: New test.

	Jakub