From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from brown.birch.relay.mailchannels.net (brown.birch.relay.mailchannels.net [23.83.209.23]) by sourceware.org (Postfix) with ESMTPS id D47F83858C2C for ; Wed, 15 Sep 2021 03:23:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org D47F83858C2C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=gotplt.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gotplt.org X-Sender-Id: dreamhost|x-authsender|siddhesh@gotplt.org Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id C95996E1415; Wed, 15 Sep 2021 03:23:32 +0000 (UTC) Received: from pdx1-sub0-mail-a40.g.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 4079F6E0C1E; Wed, 15 Sep 2021 03:23:32 +0000 (UTC) X-Sender-Id: dreamhost|x-authsender|siddhesh@gotplt.org Received: from pdx1-sub0-mail-a40.g.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384) by 100.123.194.3 (trex/6.4.3); Wed, 15 Sep 2021 03:23:32 +0000 X-MC-Relay: Junk X-MailChannels-SenderId: dreamhost|x-authsender|siddhesh@gotplt.org X-MailChannels-Auth-Id: dreamhost X-Glossy-Versed: 09f7b6d9168bd405_1631676212519_3921715437 X-MC-Loop-Signature: 1631676212518:924786859 X-MC-Ingress-Time: 1631676212518 Received: from pdx1-sub0-mail-a40.g.dreamhost.com (localhost [127.0.0.1]) by pdx1-sub0-mail-a40.g.dreamhost.com (Postfix) with ESMTP id ECEEC8BAE9; Tue, 14 Sep 2021 20:23:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gotplt.org; h=subject:to :references:from:message-id:date:mime-version:in-reply-to :content-type:content-transfer-encoding; s=gotplt.org; bh=eoA0Zm sT8Caed526jzFBCPR/Pg8=; b=pIMWciDmgrFp9QhMtOINhKE7k1AuuEK0+bg/6Y za0JsNA5WnTpZfcKa+h50cgAxPUJRYXZnuyDmrCA5UDNZasFA9B7e9jQtPBaqQ97 TICgEqZQ8jKJL0J6+iv0oJ70Q0oaJChc7fscbJKbzDhau5ogX1e68A6zhWSvzZMf RhwA4= Received: from [192.168.1.174] (unknown [1.186.224.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: siddhesh@gotplt.org) by pdx1-sub0-mail-a40.g.dreamhost.com (Postfix) with ESMTPSA id 537EA8BAE8; Tue, 14 Sep 2021 20:23:29 -0700 (PDT) Subject: Re: [PATCH v4] benchtests: Fix validate_benchout.py exceptions To: Naohiro Tamura , libc-alpha@sourceware.org References: <2f0293c7-d0b2-0a3b-1ff9-48127837d3c5@gotplt.org> <20210913134412.751102-1-naohirot@fujitsu.com> X-DH-BACKEND: pdx1-sub0-mail-a40 From: Siddhesh Poyarekar Message-ID: <6fd1bad0-61bc-f114-4218-083832a693a8@gotplt.org> Date: Wed, 15 Sep 2021 08:53:24 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210913134412.751102-1-naohirot@fujitsu.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-3039.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Sep 2021 03:23:35 -0000 On 9/13/21 7:14 PM, Naohiro Tamura wrote: > This patch fixed validate_benchout.py two exceptions, AttributeError > if benchout_strings.schema.json is specified and > json.decoder.JSONDecodeError if benchout is not JSON. > > AttributeError unconditionally occurs with a correct JSON benchout > file such as below because the code > "bench['functions'][func][k].keys()" is either "bench-variant", > "ifunc", or "results" that doesn't have keys()." > > $ ~/glibc/benchtests/scripts/validate_benchout.py bench-memcpy.out \ > ~/glibc/benchtests/scripts/benchout_strings.schema.json > Traceback (most recent call last): > File "/home/naohirot/work/github/glibc/benchtests/scripts/validate_benchout.py", line 86, in > sys.exit(main(sys.argv[1:])) > File "/home/naohirot/work/github/glibc/benchtests/scripts/validate_benchout.py", line 69, in main > bench.parse_bench(args[0], args[1]) > File "/home/naohirot/work/github/glibc/benchtests/scripts/import_bench.py", line 139, in parse_bench > do_for_all_timings(bench, lambda b, f, v: > File "/home/naohirot/work/github/glibc/benchtests/scripts/import_bench.py", line 107, in do_for_all_timings > if 'timings' not in bench['functions'][func][k].keys(): > AttributeError: 'str' object has no attribute 'keys' > > $ cat bench-memcpy.out > 1 { > 2 "timing_type": "hp_timing", > 3 "functions": { > 4 "memcpy": { > 5 "bench-variant": "default", > 6 "ifuncs": ["generic_memcpy", "__memcpy_thunderx", "__memcpy_thunderx2", "__memcpy_falkor", "__memcpy_simd", "__memcpy_a64fx", "__memcpy_generic"], > 7 "results": [ > 8 { > 9 "length": 1, > 10 "align1": 0, > 11 "align2": 0, > 12 "dst > src": 0, > 13 "timings": [10.9326, 11.0449, 11.5515, 13.5693, 11.5198, 6.77368, 11.5259] > 14 }, > ... > --- > benchtests/scripts/import_bench.py | 17 +++++++++++------ > benchtests/scripts/validate_benchout.py | 6 +++++- > 2 files changed, 16 insertions(+), 7 deletions(-) > > diff --git a/benchtests/scripts/import_bench.py b/benchtests/scripts/import_bench.py > index a799b4e1b7dc..f5e67570d4c5 100644 > --- a/benchtests/scripts/import_bench.py > +++ b/benchtests/scripts/import_bench.py > @@ -101,13 +101,18 @@ def do_for_all_timings(bench, callback): > Args: > bench: The benchmark object > callback: The callback function > + Raises: > + validator.exceptions.ValidationError: if 'timings' key not found > """ > for func in bench['functions'].keys(): > for k in bench['functions'][func].keys(): > - if 'timings' not in bench['functions'][func][k].keys(): > - continue > - > - callback(bench, func, k) > + if k == 'results': > + for r in range(len(bench['functions'][func][k])): > + if 'timings' not in bench['functions'][func][k][r].keys(): > + raise validator.exceptions.ValidationError( > + "'timings' key not found") > + else: > + callback(bench, func, k, r) > > > def compress_timings(points): > @@ -136,6 +141,6 @@ def parse_bench(filename, schema_filename): > with open(filename, 'r') as benchfile: > bench = json.load(benchfile) > validator.validate(bench, schema) > - do_for_all_timings(bench, lambda b, f, v: > - b['functions'][f][v]['timings'].sort()) > + do_for_all_timings(bench, lambda b, f, v, r: > + b['functions'][f][v][r]['timings'].sort()) > return bench This will break the original use case, i.e. bench.out, since it doesn't have the 'results' nesting timings. The sorting doesn't seem necessary for validation, it's only necessary for compare_bench.py. You could move the do_for_all_timings call into compare_bench.py, which is specific to bench.out so that you don't have to modify do_for_all_timings. Siddhesh