From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vs1-xe36.google.com (mail-vs1-xe36.google.com [IPv6:2607:f8b0:4864:20::e36]) by sourceware.org (Postfix) with ESMTPS id 1CE413858D33 for ; Thu, 16 Feb 2023 13:10:51 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1CE413858D33 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-vs1-xe36.google.com with SMTP id d66so1821157vsd.9 for ; Thu, 16 Feb 2023 05:10:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=xoyKYJBuNYmoe6qVUOJhBgjLKvCzGHjeiA4B+/E19DY=; b=TC2aUq9LHdhh8OgBbQb8+qSPa/8PsYWFORUV3IdwirxVWNthCELq5PwewyJC7XGi0/ HjGVPRo9MOhayUPi7sgn+rhK/UpA7ISh3M6I02/3+X0J1jXEwUNLaNsiC8AwGG4I7oFs kvLCULSwOVcp1u9fzO224URb3fQGkHSVnbeSvpZCwGH5Nmg+VQgZ384sduxcGNzisyVS Jl1qm1bsau89uX7VrLFmWeLkQbULyFMiDm989FTlT+59DxI9GHbk+yd7qeADyGLUhMbf hOx8NmbS2JBMijdMWStpDTbPG7g/OB7rlBe6l+cl6vHQ0fHHJgYBfm4HBejmfmAu3og8 nHKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xoyKYJBuNYmoe6qVUOJhBgjLKvCzGHjeiA4B+/E19DY=; b=7kiyjXcxikLQ6qzlbiW+MQmtYF0RHefSPgCHvFBkSKDGqa/sNPBtdpU8W8LYh/Ul4b x0JE35j95LV7I7IC77WDzG7jTMLAyAY7e12i3XZ70tU/x7V/cXloNy5gTgwxPLN1RmJI OWYkkGx0hllw5z/qlqE2tmhRajkam1wtd1aLpHDtJsh33W4Y/XlafmUaHW7ayp954aOo f/YWi9B43iY33IQgz0FoTsA4UWW50l3DqRahIpf8qKxhTLICB0WyV64z3S1n2IxfTI5q QCtxEWJGtEdP2/tRdcaahUiuOTHY/Wys1h+b0Zy7mif2kyPJcwFCW8E8OQdYhCx8sawn OcEQ== X-Gm-Message-State: AO0yUKXkRgZcbctl6lIxOwjGgjR3dTqzbfGbrONU1vg5VMenE5jci7Ge rrLd4Lvn5Dn4Ndir5Ly7gIeISvH5SUTZzAvHP9Q= X-Google-Smtp-Source: AK7set94DyaOQVMYNKRSHIxCJVqY0RPofWmyhmIlIj69u4rxJa01+xQoQa5vIHPs86fl/XxEdsuV9/6tIEPKunK1i4E= X-Received: by 2002:a67:6a82:0:b0:412:4425:d7f4 with SMTP id f124-20020a676a82000000b004124425d7f4mr1074465vsc.86.1676553050140; Thu, 16 Feb 2023 05:10:50 -0800 (PST) MIME-Version: 1.0 References: <20230216033619.16472-1-juzhe.zhong@rivai.ai> <447EFFD96B546C2E+2023021617534774493316@rivai.ai> <57BF69BEED6DBF51+2023021618313263301330@rivai.ai> In-Reply-To: <57BF69BEED6DBF51+2023021618313263301330@rivai.ai> From: Kito Cheng Date: Thu, 16 Feb 2023 21:10:39 +0800 Message-ID: Subject: Re: Re: [PATCH] RISC-V: Add vm* mask C api tests To: "juzhe.zhong@rivai.ai" Cc: jakub , gcc-patches , jeffreyalaw Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,KAM_SHORT,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: TL;DR: I think most parts of the test could be added by generator instead of adding those test cases directly, we gonna stop putting all API testing testcase. RISC-V Vector intrinsic is not implement through the *.def file way, it's using same approach as SVE's intrinsic, create and register by C files, the reason is RISC-V vector has a huge set for the intrinsic function: about ~80k function for different combinations. So that result we have so huge testcase set, and let me break down that There are several kinds for those testcase: 1. vsetvli insertion pass testing, which is a highly customized mode switching pass 2. Code gen test: testing our move pattern and generated code has satisfied the RISC-V vector ISA constraint. 3. Intrinsic API testing: test the C intrinsic has right interface and generated expected instruction --- FIrst part has 375 testcase in `testsuite/gcc.target/riscv/rvv/vsetvl` which is important and ~16M but one potential issue is that is highly code gen sensitive, we've added many long scan-assembly in the test file, it's not ideal and we plan to implement a builtin verifier inside GCC instead of lots of long scan-assembly, This is planned for this year, but will happen after GCC 13 release. --- Second part is also important, and only 300~400 files, so I think this part should just keep as it is. --- The last part is the most huge part in the testcases (~3000 files so far), and I think we should consider removing this part from the GCC testsuite, since we have a standard one[1] from the RISC-V international. So my thought is we stop putting further intrinsic API testing now, and use the external one, and evaluate the effort and benefit of implementing a test generator inside GCC in future (after GCC 13 release). [1] https://github.com/riscv-non-isa/rvv-intrinsic-doc/blob/master/auto-generated/api-testing/ On Thu, Feb 16, 2023 at 6:32 PM juzhe.zhong@rivai.ai wrote: > > Well, I think the best solution: > 1. Remove all intrinsic test that I already commited. > 2. Then, embed test-generator for this intrinsic unit-test. > 3. Call test-generator during regression and test them. > 4. Remove the testcases generated by the test-generator after regression. > > Not sure whether you aggree with me. > > The test-generator I used is generating the testcase by reading the rvv-intrinsic document directly and generate the testcases. > That means I need to commit test-generator and rvv-intrinsic document both. > I don't think my test-generator is good to commit. > > I believe Kito has the mature and better test-generator (much better than mine) to commit since rvv-intrinsic doc is their work. > > As long as we can make kito's test-generator embedded into GCC regression, this issue will be fixed. And I believe we can fix it soon. > > So...Let's wait for kito. > > > juzhe.zhong@rivai.ai > > From: Jakub Jelinek > Date: 2023-02-16 18:20 > To: juzhe.zhong@rivai.ai > CC: gcc-patches; kito.cheng; jeffreyalaw > Subject: Re: Re: [PATCH] RISC-V: Add vm* mask C api tests > On Thu, Feb 16, 2023 at 05:53:48PM +0800, juzhe.zhong@rivai.ai wrote: > > Thanks for reporting this. I think may be we can make reduce tests into 1/3. > > For example: > > We have: > > * gcc.target/riscv/rvv/base/vmand_mm-1.c: New test. > > * gcc.target/riscv/rvv/base/vmand_mm-2.c: New test. > > * gcc.target/riscv/rvv/base/vmand_mm-3.c: New test. > > > > Maybe we can reduce it into one test: > > vmand_mm.c only. > > > > I will improve and reduce all intrinsic tests like this soon (I almost done all intrinsic in this week, next week I will do this soon). > > > > RVV intrinsics are really huge, this is the document: > > https://github.com/riscv-non-isa/rvv-intrinsic-doc/tree/master/auto-generated > > > > The testcases are directly come from LLVM (We just add assembler check into the test), they also have this amount of testcases and the just recently change them: > > https://reviews.llvm.org/D142697 > > https://reviews.llvm.org/D142644 > > > > Take a look at the changing LLVM patch, I am aggree with you ,the LLVM patch is quite huge and not easy to maintain. > > Yeah, LLVM does this all the time, their unit-tests where they embed e.g. > matchers for IL in huge tests. > > I just think the way they are doing this is a very bad idea. > If say one writes some C/C++ test, compile it, some helper program > adds the IL into comments in the test then again any time you want to > adjust something in the compiler that affects those tests, you need to > regenerate them. Is the generator included somewhere, or does every > user write his own tooling to do that? Anyway, if the solution is > regenerate the IL, the test lost quite lot of its meaning, because > when changing thousands of tests and regenerating the IL for all of them, > one can hardly expect to carefully examine the changes to all those tests > whether everything was intended. > > In GCC we have far fewer such unit-tests and big parts of the testsuite > are testing everything from parsing through assembly through linking through > runtime. In my experience over the years, many such tests can discover even > bugs completely unrelated to the original reason why a test has been added. > > If they have some generator in LLVM for these riscv tests, even worse, > there is another step for LLVM generator regenerates them on the LLVM side > and somebody needs to reimport them into GCC and regenerate the > scan-assembler regexps. > > riscv already uses what various other GCC backends use for builtins and > intrinsics, various *.def files from which the actual support is created. > So, can't we use the same files + something on top of that to have the > testsuite coverage, or if it should be independent from it, at least > have something similar which would describe intrinsic that should be tested, > iterate over such and such types for which arguments and how to come up with > the expected emitted code. > So, rather than reducing the tests into 1/3, try to reduce them to one > line per intrinsic or something of that scale. > > Jakub > >