From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by sourceware.org (Postfix) with ESMTPS id 11EF13858400 for ; Mon, 18 Oct 2021 12:57:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 11EF13858400 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 19IClh3n005316; Mon, 18 Oct 2021 08:57:17 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3bs1u59vav-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 18 Oct 2021 08:57:17 -0400 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 19ICnZPD005364; Mon, 18 Oct 2021 08:57:16 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 3bs1u59vag-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 18 Oct 2021 08:57:16 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 19ICruEW028291; Mon, 18 Oct 2021 12:57:15 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma01wdc.us.ibm.com with ESMTP id 3bqpca53fg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 18 Oct 2021 12:57:15 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 19ICvFbH37159226 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Oct 2021 12:57:15 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 39E55112061; Mon, 18 Oct 2021 12:57:15 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A8673112062; Mon, 18 Oct 2021 12:57:14 +0000 (GMT) Received: from localhost (unknown [9.160.105.200]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 18 Oct 2021 12:57:14 +0000 (GMT) Content-Type: text/plain; charset="utf-8" In-Reply-To: References: <20210805074733.433430-1-naohirot@fujitsu.com> <163154191414.705584.12050866556951422556@localhost.localdomain> Subject: Re: [PATCH v3 2/5] benchtests: Add memset zero fill benchtest From: "Lucas A. M. Magalhaes" To: libc-alpha@sourceware.org, naohirot@fujitsu.com Cc: "'Wilco Dijkstra'" , Noah Goldstein Date: Mon, 18 Oct 2021 09:57:12 -0300 Message-ID: <163456183281.2142698.11944761470468149892@localhost.localdomain> User-Agent: alot/0.9.1 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: YuWjw_yQmLEJnw1np0j4ks2w-ZigjjgD X-Proofpoint-ORIG-GUID: x48X312xMkM5Ve_CELRlLWxmAKmZz9xi Content-Transfer-Encoding: quoted-printable X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.0.607.475 definitions=2021-10-18_05,2021-10-14_02,2020-04-07_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 impostorscore=0 mlxlogscore=999 bulkscore=0 spamscore=0 clxscore=1011 adultscore=0 suspectscore=0 mlxscore=0 phishscore=0 malwarescore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001 definitions=main-2110180077 X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, KAM_SHORT, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Oct 2021 12:57:23 -0000 > > > What do you think? > > > > As I've mentioned, this will never work using the current benchmark loo= p. > > At size 256 your loop has only 1 timer tick... The only way to get any = data > > out is to increase the time taken per call. At 16K there are about 20 t= icks so > > it is still very inaccurate. By repeating the test thousands of times y= ou can > > some signal out (eg. 20% is 20 ticks, 80% is 21 gives ~20.8 ticks on av= erage), > > but that's impossible for smaller sizes. > > > > So if you want to measure small sizes, you need to use a more accurate = timing > > loop. >=20 > Thank you for the comment. > OK, I understood. So I updated the start size to 16KB too to commit first. > Please find V5 [1] and merge it if it's OK. > Changes from V4: > - Start size to 16KB from 256B > - End size to 16MB from 64MB =20 > [1] https://sourceware.org/pipermail/libc-alpha/2021-September/131245.html =20 Hi Tamura, I agree with you that is important to measure calls with smaller lengths. IMHO the issue here is not if the benchmark should measure or not this lengths, but how it could measure that. +static void +__attribute__((noinline, noclone)) +do_one_test (json_ctx_t *json_ctx, impl_t *impl, CHAR *s, + int c1 __attribute ((unused)), int c2 __attribute ((unused)), + size_t n) +{ + size_t i, iters =3D 32; + timing_t start, stop, cur, latency =3D 0; + + CALL (impl, s, c2, n); // warm up + + for (i =3D 0; i < iters; i++) + { + memset (s, c1, n); // alternation + + TIMING_NOW (start); + + CALL (impl, s, c2, n); + + TIMING_NOW (stop); + TIMING_DIFF (cur, start, stop); + TIMING_ACCUM (latency, cur); + } + + json_element_double (json_ctx, (double) latency / (double) iters); +} By doing this you are measuring just the call it self and accumulating the results. This is indeed not measurable for really small lengths. You could try moving the memset and the timing out of the loop and measure the time spent in multiple runs. To fix the memset you could memset a bigger buffer and move the s pointer on each loop. I guess this will reduce the variations Wilco mentioned. Maybe we need to keep this loop for bigger lengths as we will need a buffer too much big for the implementation that I suggested. Another point here is that GNU Code Style asks for /**/ comments instead of //. As seen in http://www.gnu.org/prep/standards/standards.html#Comments Finally, Sorry that I took so long to reply here. Thanks for working on this. --- Lucas A. M. Magalh=C3=A3es