Well, I think the best solution:
1. Remove all intrinsic test that I already commited.
2. Then, embed test-generator for this intrinsic unit-test.
3. Call  test-generator during regression and test them.
4. Remove the testcases generated by the test-generator after regression.

Not sure whether you aggree with me.

The test-generator I used is generating the testcase by reading the rvv-intrinsic document directly and generate the testcases.
That means I need to commit test-generator and rvv-intrinsic document both.
I don't think my test-generator is good to commit.

I believe Kito has the mature and better test-generator (much better than mine) to commit since rvv-intrinsic doc is their work.

As long as we can make kito's test-generator embedded into GCC regression, this issue will be fixed. And I believe we can fix it soon.

So...Let's wait for kito.


juzhe.zhong@rivai.ai
 
From: Jakub Jelinek
Date: 2023-02-16 18:20
To: juzhe.zhong@rivai.ai
CC: gcc-patches; kito.cheng; jeffreyalaw
Subject: Re: Re: [PATCH] RISC-V: Add vm* mask C api tests
On Thu, Feb 16, 2023 at 05:53:48PM +0800, juzhe.zhong@rivai.ai wrote:
> Thanks for reporting this. I think may be we can make reduce tests into 1/3.
> For example:
> We have:
> * gcc.target/riscv/rvv/base/vmand_mm-1.c: New test.
> * gcc.target/riscv/rvv/base/vmand_mm-2.c: New test.
> * gcc.target/riscv/rvv/base/vmand_mm-3.c: New test.
> 
> Maybe we can reduce it into one test:
> vmand_mm.c only.
> 
> I will improve and reduce all intrinsic tests like this soon (I almost done all intrinsic in this week, next week I will do this soon).
> 
> RVV intrinsics are really huge, this is the document:
> https://github.com/riscv-non-isa/rvv-intrinsic-doc/tree/master/auto-generated 
> 
> The testcases are directly come from LLVM (We just add assembler check into the test), they also have this amount of testcases and the just recently change them:
> https://reviews.llvm.org/D142697 
> https://reviews.llvm.org/D142644 
> 
> Take a look at the changing LLVM patch, I am aggree with you ,the LLVM patch is quite huge and not easy to maintain.
 
Yeah, LLVM does this all the time, their unit-tests where they embed e.g.
matchers for IL in huge tests.
 
I just think the way they are doing this is a very bad idea.
If say one writes some C/C++ test, compile it, some helper program
adds the IL into comments in the test then again any time you want to
adjust something in the compiler that affects those tests, you need to
regenerate them.  Is the generator included somewhere, or does every
user write his own tooling to do that?  Anyway, if the solution is
regenerate the IL, the test lost quite lot of its meaning, because
when changing thousands of tests and regenerating the IL for all of them,
one can hardly expect to carefully examine the changes to all those tests
whether everything was intended.
 
In GCC we have far fewer such unit-tests and big parts of the testsuite
are testing everything from parsing through assembly through linking through
runtime.  In my experience over the years, many such tests can discover even
bugs completely unrelated to the original reason why a test has been added.
 
If they have some generator in LLVM for these riscv tests, even worse,
there is another step for LLVM generator regenerates them on the LLVM side
and somebody needs to reimport them into GCC and regenerate the
scan-assembler regexps.
 
riscv already uses what various other GCC backends use for builtins and
intrinsics, various *.def files from which the actual support is created.
So, can't we use the same files + something on top of that to have the
testsuite coverage, or if it should be independent from it, at least
have something similar which would describe intrinsic that should be tested,
iterate over such and such types for which arguments and how to come up with
the expected emitted code.
So, rather than reducing the tests into 1/3, try to reduce them to one
line per intrinsic or something of that scale.
 
Jakub