* [PATCH 1/3] benchtests: Wide range of tests names in schema.json
@ 2017-11-16 22:54 Victor Rodriguez
2017-11-16 22:54 ` [PATCH 2/3] benchtests: Adjust valid and accepted properties Victor Rodriguez
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Victor Rodriguez @ 2017-11-16 22:54 UTC (permalink / raw)
To: libc-alpha; +Cc: Victor Rodriguez
When executing bench-math the benchmark output is invalid with this
error msg:
Invalid benchmark output: 'workload-spec2006.wrf' does not match any of
the regexes: '^[_a-zA-Z0-9]*$¹ or Invalid benchmark output: Additional
properties are not allowed ('workload-spec2006.wrf' was unexpected)
This patch change regex in benchout.schema.json to accept simbols in
benchmark tests names
(VERSION): Set to 2.27
* benchtests/scripts/benchout.schema.json: Fix regex to accept a
wide range of tests names
Signed-off-by: Victor Rodriguez <victor.rodriguez.bahena@intel.com>
---
ChangeLog | 6 ++++++
benchtests/scripts/benchout.schema.json | 2 +-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/ChangeLog b/ChangeLog
index 8dbfc7e..7d0fd63 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,9 @@
+2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
+
+ (VERSION): Set to 2.27
+ * benchtests/scripts/benchout.schema.json: Fix regex to accept a wide range
+ of tests names.
+
2017-08-02 Siddhesh Poyarekar <siddhesh@sourceware.org>
* version.h (RELEASE): Set to "stable"
diff --git a/benchtests/scripts/benchout.schema.json b/benchtests/scripts/benchout.schema.json
index affb7c1..814c633 100644
--- a/benchtests/scripts/benchout.schema.json
+++ b/benchtests/scripts/benchout.schema.json
@@ -13,7 +13,7 @@
"title": "Function names",
"type": "object",
"patternProperties": {
- "^[_a-zA-Z0-9]*$": {
+ "^[_a-zA-Z0-9_,=.-]*$": {
"title": "Function variants",
"type": "object",
"properties": {
--
2.15.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 3/3] benchtests:Enable BENCHSET to run subset of tests
2017-11-16 22:54 [PATCH 1/3] benchtests: Wide range of tests names in schema.json Victor Rodriguez
2017-11-16 22:54 ` [PATCH 2/3] benchtests: Adjust valid and accepted properties Victor Rodriguez
@ 2017-11-16 22:54 ` Victor Rodriguez
2017-11-22 10:34 ` Siddhesh Poyarekar
2017-11-17 1:12 ` [PATCH 1/3] benchtests: Wide range of tests names in schema.json Paul Clarke
2 siblings, 1 reply; 7+ messages in thread
From: Victor Rodriguez @ 2017-11-16 22:54 UTC (permalink / raw)
To: libc-alpha; +Cc: Victor Rodriguez, Icarus Sparry
This patch adds BENCHSET variable to benchtests/Makefile in order to
provide the capability to run a list of subsets of benchmark tests, ie;
make bench BENCHSET="bench-pthread bench-math malloc-thread"
This Helps users to benchmark specific glibc area
Changelog:
2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
(VERSION): Set to 2.27
* benchtests/Makefile:Add BENCHSET to allow subsets of
benchmarks to be run.
* benchtests/README: Add documentation for: Running subsets of
benchmarks.
Signed-off-by: Victor Rodriguez <victor.rodriguez.bahena@intel.com>
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
---
ChangeLog | 9 +++++++
benchtests/Makefile | 69 +++++++++++++++++++++++++++++++++++++----------------
benchtests/README | 20 ++++++++++++++++
3 files changed, 78 insertions(+), 20 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index 6a6bb62..bf61c72 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,12 @@
+Changelog:
+2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
+
+ (VERSION): Set to 2.27
+ * benchtests/Makefile:Add BENCHSET to allow subsets of
+ benchmarks to be run
+ * benchtests/README: Add documentation for: Running subsets of
+ benchmarks.
+
2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
(VERSION): Set to 2.27
diff --git a/benchtests/Makefile b/benchtests/Makefile
index 37788e8..6712710 100644
--- a/benchtests/Makefile
+++ b/benchtests/Makefile
@@ -29,7 +29,11 @@ bench-pthread := pthread_once thread_create
bench-string := ffs ffsll
+ifeq (${BENCHSET},)
bench := $(bench-math) $(bench-pthread) $(bench-string)
+else
+bench := $(foreach B,$(filter bench-%,${BENCHSET}), ${${B}})
+endif
# String function benchmarks.
string-benchset := bcopy bzero memccpy memchr memcmp memcpy memmem memmove \
@@ -66,8 +70,12 @@ stdio-common-benchset := sprintf
math-benchset := math-inlines
+ifeq (${BENCHSET},)
benchset := $(string-benchset-all) $(stdlib-benchset) $(stdio-common-benchset) \
$(math-benchset)
+else
+benchset := $(foreach B,$(filter %-benchset,${BENCHSET}), ${${B}})
+endif
CFLAGS-bench-ffs.c += -fno-builtin
CFLAGS-bench-ffsll.c += -fno-builtin
@@ -77,7 +85,11 @@ CFLAGS-bench-fminf.c += -fno-builtin
CFLAGS-bench-fmax.c += -fno-builtin
CFLAGS-bench-fmaxf.c += -fno-builtin
+ifeq (${BENCHSET},)
bench-malloc := malloc-thread
+else
+bench-malloc := $(filter malloc-%,${BENCHSET})
+endif
$(addprefix $(objpfx)bench-,$(bench-math)): $(libm)
$(addprefix $(objpfx)bench-,$(math-benchset)): $(libm)
@@ -144,6 +156,19 @@ bench-clean:
rm -f $(timing-type) $(addsuffix .o,$(timing-type))
rm -f $(addprefix $(objpfx),$(bench-extra-objs))
+# Validate the passed in BENCHSET
+ifneq ($(strip ${BENCHSET}),)
+VALIDBENCHSETNAMES := bench-pthread bench-math bench-string string-benchset \
+ wcsmbs-benchset stdlib-benchset stdio-common-benchset math-benchset \
+ malloc-thread
+INVALIDBENCHSETNAMES := $(filter-out ${VALIDBENCHSETNAMES},${BENCHSET})
+ifneq (${INVALIDBENCHSETNAMES},)
+$(info The following values in BENCHSET are invalid: ${INVALIDBENCHSETNAMES})
+$(info The valid ones are: ${VALIDBENCHSETNAMES})
+$(error Invalid BENCHSET value)
+endif
+endif
+
# Define the bench target only if the target has a usable python installation.
ifdef PYTHON
bench: bench-build bench-set bench-func bench-malloc
@@ -170,10 +195,11 @@ bench-set: $(binaries-benchset)
done
bench-malloc: $(binaries-bench-malloc)
- run=$(objpfx)bench-malloc-thread; \
- for thr in 1 8 16 32; do \
- echo "Running $${run} $${thr}"; \
+ for run in $^; do \
+ for thr in 1 8 16 32; do \
+ echo "Running $${run} $${thr}"; \
$(run-bench) $${thr} > $${run}-$${thr}.out; \
+ done;\
done
# Build and execute the benchmark functions. This target generates JSON
@@ -181,25 +207,28 @@ bench-malloc: $(binaries-bench-malloc)
# so one could even execute them individually and process it using any JSON
# capable language or tool.
bench-func: $(binaries-bench)
+ if [ -n '$^' ] ; then \
{ timing_type=$$($(timing-type)); \
- echo "{\"timing_type\": \"$${timing_type}\","; \
- echo " \"functions\": {"; \
- for run in $^; do \
- if ! [ "x$${run}" = "x$<" ]; then \
- echo ","; \
+ echo "{\"timing_type\": \"$${timing_type}\","; \
+ echo " \"functions\": {"; \
+ for run in $^; do \
+ if ! [ "x$${run}" = "x$<" ]; then \
+ echo ","; \
+ fi; \
+ echo "Running $${run}" >&2; \
+ $(run-bench) $(DETAILED_OPT); \
+ done; \
+ echo; \
+ echo " }"; \
+ echo "}"; \
+ } > $(objpfx)bench.out-tmp; \
+ if [ -f $(objpfx)bench.out ]; then \
+ mv -f $(objpfx)bench.out $(objpfx)bench.out.old; \
fi; \
- echo "Running $${run}" >&2; \
- $(run-bench) $(DETAILED_OPT); \
- done; \
- echo; \
- echo " }"; \
- echo "}"; } > $(objpfx)bench.out-tmp; \
- if [ -f $(objpfx)bench.out ]; then \
- mv -f $(objpfx)bench.out $(objpfx)bench.out.old; \
- fi; \
- mv -f $(objpfx)bench.out-tmp $(objpfx)bench.out
- $(PYTHON) scripts/validate_benchout.py $(objpfx)bench.out \
- scripts/benchout.schema.json
+ mv -f $(objpfx)bench.out-tmp $(objpfx)bench.out; \
+ $(PYTHON) scripts/validate_benchout.py $(objpfx)bench.out \
+ scripts/benchout.schema.json; \
+ fi
$(timing-type) $(binaries-bench) $(binaries-benchset) \
$(binaries-bench-malloc): %: %.o $(objpfx)json-lib.o \
diff --git a/benchtests/README b/benchtests/README
index b015acf..20942c7 100644
--- a/benchtests/README
+++ b/benchtests/README
@@ -53,6 +53,26 @@ otherwise the above command may try to build the benchmark again. Benchmarks
that require generated code to be executed during the build are skipped when
cross-building.
+Running subsets of benchmarks:
+==============================
+
+If a subset of benchmarks needs to be run one may run the benchmark by invoking
+make as follows:
+
+ $ make bench BENCHSET="bench-pthread bench-math malloc-thread"
+
+Allowing to measure the performance of specific glibc area. Posible areas are:
+
+ bench-math
+ bench-pthread
+ bench-string
+ string-benchset
+ wcsmbs-benchset
+ stdlib-benchset
+ stdio-common-benchset
+ math-benchset
+ malloc-thread
+
Adding a function to benchtests:
===============================
--
2.15.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/3] benchtests: Adjust valid and accepted properties
2017-11-16 22:54 [PATCH 1/3] benchtests: Wide range of tests names in schema.json Victor Rodriguez
@ 2017-11-16 22:54 ` Victor Rodriguez
2017-11-22 10:25 ` Siddhesh Poyarekar
2017-11-16 22:54 ` [PATCH 3/3] benchtests:Enable BENCHSET to run subset of tests Victor Rodriguez
2017-11-17 1:12 ` [PATCH 1/3] benchtests: Wide range of tests names in schema.json Paul Clarke
2 siblings, 1 reply; 7+ messages in thread
From: Victor Rodriguez @ 2017-11-16 22:54 UTC (permalink / raw)
To: libc-alpha; +Cc: Victor Rodriguez
Benchmark workload-spec2006.wrf does not produce max, min or mean
results but instead produce throughput. This is represented in
benchtests/bench-skeleton.c. This patch adjust benchout.schema.json to consider
bench.out from bench-math benchmarks as valid
2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
(VERSION): Set to 2.27
* benchtests/scripts/benchout.schema.json: Add throughput as accepted
result from property and remove "max", min" and "mean" from required
properties based on benchtests/bench-skeleton.c.
Signed-off-by: Victor Rodriguez <victor.rodriguez.bahena@intel.com>
---
ChangeLog | 7 +++++++
benchtests/scripts/benchout.schema.json | 3 ++-
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/ChangeLog b/ChangeLog
index 7d0fd63..6a6bb62 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
+
+ (VERSION): Set to 2.27
+ * benchtests/scripts/benchout.schema.json: Add throughput as accepted
+ result from property and remove "max", min" and "mean" from required
+ properties based on benchtests/bench-skeleton.c.
+
2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
(VERSION): Set to 2.27
diff --git a/benchtests/scripts/benchout.schema.json b/benchtests/scripts/benchout.schema.json
index 814c633..7edd2a7 100644
--- a/benchtests/scripts/benchout.schema.json
+++ b/benchtests/scripts/benchout.schema.json
@@ -19,6 +19,7 @@
"properties": {
"duration": {"type": "number"},
"iterations": {"type": "number"},
+ "throughput": {"type": "number"},
"max": {"type": "number"},
"min": {"type": "number"},
"mean": {"type": "number"},
@@ -27,7 +28,7 @@
"items": {"type": "number"}
}
},
- "required": ["duration", "iterations", "max", "min", "mean"],
+ "required": ["duration", "iterations"],
"additionalProperties": false
}
},
--
2.15.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/3] benchtests: Wide range of tests names in schema.json
2017-11-16 22:54 [PATCH 1/3] benchtests: Wide range of tests names in schema.json Victor Rodriguez
2017-11-16 22:54 ` [PATCH 2/3] benchtests: Adjust valid and accepted properties Victor Rodriguez
2017-11-16 22:54 ` [PATCH 3/3] benchtests:Enable BENCHSET to run subset of tests Victor Rodriguez
@ 2017-11-17 1:12 ` Paul Clarke
2017-11-18 14:24 ` Rodriguez Bahena, Victor
2 siblings, 1 reply; 7+ messages in thread
From: Paul Clarke @ 2017-11-17 1:12 UTC (permalink / raw)
To: Victor Rodriguez, libc-alpha
On 11/16/2017 04:52 PM, Victor Rodriguez wrote:
> When executing bench-math the benchmark output is invalid with this
> error msg:
>
> Invalid benchmark output: 'workload-spec2006.wrf' does not match any of
> the regexes: '^[_a-zA-Z0-9]*$¹ or Invalid benchmark output: Additional
> properties are not allowed ('workload-spec2006.wrf' was unexpected)
>
> This patch change regex in benchout.schema.json to accept simbols in
s/change/changes/
s/simbols/symbols/
> benchmark tests names
>
> (VERSION): Set to 2.27
> * benchtests/scripts/benchout.schema.json: Fix regex to accept a
> wide range of tests names
s/wide/wider/
Also, should the subject be "Widen" (or "Expand"?), instead of "Wide" ?
>
> Signed-off-by: Victor Rodriguez <victor.rodriguez.bahena@intel.com>
> ---
> ChangeLog | 6 ++++++
> benchtests/scripts/benchout.schema.json | 2 +-
> 2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/ChangeLog b/ChangeLog
> index 8dbfc7e..7d0fd63 100644
> --- a/ChangeLog
> +++ b/ChangeLog
> @@ -1,3 +1,9 @@
> +2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
> +
> + (VERSION): Set to 2.27
> + * benchtests/scripts/benchout.schema.json: Fix regex to accept a wide range
s/wide/wider/?
> + of tests names.
> +
> 2017-08-02 Siddhesh Poyarekar <siddhesh@sourceware.org>
>
> * version.h (RELEASE): Set to "stable"
> diff --git a/benchtests/scripts/benchout.schema.json b/benchtests/scripts/benchout.schema.json
> index affb7c1..814c633 100644
> --- a/benchtests/scripts/benchout.schema.json
> +++ b/benchtests/scripts/benchout.schema.json
> @@ -13,7 +13,7 @@
> "title": "Function names",
> "type": "object",
> "patternProperties": {
> - "^[_a-zA-Z0-9]*$": {
> + "^[_a-zA-Z0-9_,=.-]*$": {
^ ^
It looks like '_' is there twice.
How did you come up with the set of new characters? It would be helpful to include that in the patch description.
PC
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/3] benchtests: Wide range of tests names in schema.json
2017-11-17 1:12 ` [PATCH 1/3] benchtests: Wide range of tests names in schema.json Paul Clarke
@ 2017-11-18 14:24 ` Rodriguez Bahena, Victor
0 siblings, 0 replies; 7+ messages in thread
From: Rodriguez Bahena, Victor @ 2017-11-18 14:24 UTC (permalink / raw)
To: Paul Clarke, libc-alpha
-----Original Message-----
From: Paul Clarke <pc@us.ibm.com>
Date: Thursday, November 16, 2017 at 7:12 PM
To: Victor Rodriguez Bahena <victor.rodriguez.bahena@intel.com>,
"libc-alpha@sourceware.org" <libc-alpha@sourceware.org>
Subject: Re: [PATCH 1/3] benchtests: Wide range of tests names in
schema.json
>
>
>On 11/16/2017 04:52 PM, Victor Rodriguez wrote:
>> When executing bench-math the benchmark output is invalid with this
>> error msg:
>>
>> Invalid benchmark output: 'workload-spec2006.wrf' does not match any of
>> the regexes: '^[_a-zA-Z0-9]*$¹ or Invalid benchmark output: Additional
>> properties are not allowed ('workload-spec2006.wrf' was unexpected)
>>
>> This patch change regex in benchout.schema.json to accept simbols in
>
>s/change/changes/
>s/simbols/symbols/
>
>> benchmark tests names
>>
>> (VERSION): Set to 2.27
>> * benchtests/scripts/benchout.schema.json: Fix regex to accept a
>> wide range of tests names
>
>s/wide/wider/
>
>Also, should the subject be "Widen" (or "Expand"?), instead of "Wide" ?
>
>>
>> Signed-off-by: Victor Rodriguez <victor.rodriguez.bahena@intel.com>
>> ---
>> ChangeLog | 6 ++++++
>> benchtests/scripts/benchout.schema.json | 2 +-
>> 2 files changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/ChangeLog b/ChangeLog
>> index 8dbfc7e..7d0fd63 100644
>> --- a/ChangeLog
>> +++ b/ChangeLog
>> @@ -1,3 +1,9 @@
>> +2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
>> +
>> + (VERSION): Set to 2.27
>> + * benchtests/scripts/benchout.schema.json: Fix regex to accept a wide
>>range
>
>s/wide/wider/?
>
>> + of tests names.
>> +
>> 2017-08-02 Siddhesh Poyarekar <siddhesh@sourceware.org>
>>
>> * version.h (RELEASE): Set to "stable"
>> diff --git a/benchtests/scripts/benchout.schema.json
>>b/benchtests/scripts/benchout.schema.json
>> index affb7c1..814c633 100644
>> --- a/benchtests/scripts/benchout.schema.json
>> +++ b/benchtests/scripts/benchout.schema.json
>> @@ -13,7 +13,7 @@
>> "title": "Function names",
>> "type": "object",
>> "patternProperties": {
>> - "^[_a-zA-Z0-9]*$": {
>> + "^[_a-zA-Z0-9_,=.-]*$": {
> ^ ^
>It looks like '_' is there twice.
Thanks , you are right
>
>How did you come up with the set of new characters? It would be helpful
>to include that in the patch description.
Test names like workload-spec2006.wrf break the script due to . and -
I will include that in the patch description and send V2
Regards
>
>PC
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 2/3] benchtests: Adjust valid and accepted properties
2017-11-16 22:54 ` [PATCH 2/3] benchtests: Adjust valid and accepted properties Victor Rodriguez
@ 2017-11-22 10:25 ` Siddhesh Poyarekar
0 siblings, 0 replies; 7+ messages in thread
From: Siddhesh Poyarekar @ 2017-11-22 10:25 UTC (permalink / raw)
To: Victor Rodriguez, libc-alpha
This is OK. But why do you have this:
(VERSION): Set to 2.27
on top of every ChangeLog entry? Please remove that.
Siddhesh
On Friday 17 November 2017 04:22 AM, Victor Rodriguez wrote:
> Benchmark workload-spec2006.wrf does not produce max, min or mean
> results but instead produce throughput. This is represented in
> benchtests/bench-skeleton.c. This patch adjust benchout.schema.json to consider
> bench.out from bench-math benchmarks as valid
>
> 2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
>
> (VERSION): Set to 2.27
> * benchtests/scripts/benchout.schema.json: Add throughput as accepted
> result from property and remove "max", min" and "mean" from required
> properties based on benchtests/bench-skeleton.c.
>
> Signed-off-by: Victor Rodriguez <victor.rodriguez.bahena@intel.com>
> ---
> ChangeLog | 7 +++++++
> benchtests/scripts/benchout.schema.json | 3 ++-
> 2 files changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/ChangeLog b/ChangeLog
> index 7d0fd63..6a6bb62 100644
> --- a/ChangeLog
> +++ b/ChangeLog
> @@ -1,3 +1,10 @@
> +2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
> +
> + (VERSION): Set to 2.27
> + * benchtests/scripts/benchout.schema.json: Add throughput as accepted
> + result from property and remove "max", min" and "mean" from required
> + properties based on benchtests/bench-skeleton.c.
> +
> 2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
>
> (VERSION): Set to 2.27
> diff --git a/benchtests/scripts/benchout.schema.json b/benchtests/scripts/benchout.schema.json
> index 814c633..7edd2a7 100644
> --- a/benchtests/scripts/benchout.schema.json
> +++ b/benchtests/scripts/benchout.schema.json
> @@ -19,6 +19,7 @@
> "properties": {
> "duration": {"type": "number"},
> "iterations": {"type": "number"},
> + "throughput": {"type": "number"},
> "max": {"type": "number"},
> "min": {"type": "number"},
> "mean": {"type": "number"},
> @@ -27,7 +28,7 @@
> "items": {"type": "number"}
> }
> },
> - "required": ["duration", "iterations", "max", "min", "mean"],
> + "required": ["duration", "iterations"],
> "additionalProperties": false
> }
> },
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 3/3] benchtests:Enable BENCHSET to run subset of tests
2017-11-16 22:54 ` [PATCH 3/3] benchtests:Enable BENCHSET to run subset of tests Victor Rodriguez
@ 2017-11-22 10:34 ` Siddhesh Poyarekar
0 siblings, 0 replies; 7+ messages in thread
From: Siddhesh Poyarekar @ 2017-11-22 10:34 UTC (permalink / raw)
To: Victor Rodriguez, libc-alpha; +Cc: Icarus Sparry
Hi,
Almost there, comments inline. Please post the final version with those
changes.
Thanks,
Siddhesh
On Friday 17 November 2017 04:22 AM, Victor Rodriguez wrote:
> This patch adds BENCHSET variable to benchtests/Makefile in order to
> provide the capability to run a list of subsets of benchmark tests, ie;
>
> make bench BENCHSET="bench-pthread bench-math malloc-thread"
>
> This Helps users to benchmark specific glibc area
>
> Changelog:
> 2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
>
> (VERSION): Set to 2.27
> * benchtests/Makefile:Add BENCHSET to allow subsets of
> benchmarks to be run.
> * benchtests/README: Add documentation for: Running subsets of
> benchmarks.
>
> Signed-off-by: Victor Rodriguez <victor.rodriguez.bahena@intel.com>
> Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
> ---
> ChangeLog | 9 +++++++
> benchtests/Makefile | 69 +++++++++++++++++++++++++++++++++++++----------------
> benchtests/README | 20 ++++++++++++++++
> 3 files changed, 78 insertions(+), 20 deletions(-)
>
> diff --git a/ChangeLog b/ChangeLog
> index 6a6bb62..bf61c72 100644
> --- a/ChangeLog
> +++ b/ChangeLog
> @@ -1,3 +1,12 @@
> +Changelog:
> +2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
> +
> + (VERSION): Set to 2.27
What is this line for?
> + * benchtests/Makefile:Add BENCHSET to allow subsets of
> + benchmarks to be run
> + * benchtests/README: Add documentation for: Running subsets of
> + benchmarks.
> +
> 2017-11-15 Victor Rodriguez <victor.rodriguez.bahena@intel.com>
>
> (VERSION): Set to 2.27
> diff --git a/benchtests/Makefile b/benchtests/Makefile
> index 37788e8..6712710 100644
> --- a/benchtests/Makefile
> +++ b/benchtests/Makefile
> @@ -29,7 +29,11 @@ bench-pthread := pthread_once thread_create
>
> bench-string := ffs ffsll
>
> +ifeq (${BENCHSET},)
> bench := $(bench-math) $(bench-pthread) $(bench-string)
> +else
> +bench := $(foreach B,$(filter bench-%,${BENCHSET}), ${${B}})
> +endif
OK.
>
> # String function benchmarks.
> string-benchset := bcopy bzero memccpy memchr memcmp memcpy memmem memmove \
> @@ -66,8 +70,12 @@ stdio-common-benchset := sprintf
>
> math-benchset := math-inlines
>
> +ifeq (${BENCHSET},)
> benchset := $(string-benchset-all) $(stdlib-benchset) $(stdio-common-benchset) \
> $(math-benchset)
> +else
> +benchset := $(foreach B,$(filter %-benchset,${BENCHSET}), ${${B}})
> +endif
OK.
>
> CFLAGS-bench-ffs.c += -fno-builtin
> CFLAGS-bench-ffsll.c += -fno-builtin
> @@ -77,7 +85,11 @@ CFLAGS-bench-fminf.c += -fno-builtin
> CFLAGS-bench-fmax.c += -fno-builtin
> CFLAGS-bench-fmaxf.c += -fno-builtin
>
> +ifeq (${BENCHSET},)
> bench-malloc := malloc-thread
> +else
> +bench-malloc := $(filter malloc-%,${BENCHSET})
> +endif
OK.
>
> $(addprefix $(objpfx)bench-,$(bench-math)): $(libm)
> $(addprefix $(objpfx)bench-,$(math-benchset)): $(libm)
> @@ -144,6 +156,19 @@ bench-clean:
> rm -f $(timing-type) $(addsuffix .o,$(timing-type))
> rm -f $(addprefix $(objpfx),$(bench-extra-objs))
>
> +# Validate the passed in BENCHSET
> +ifneq ($(strip ${BENCHSET}),)
> +VALIDBENCHSETNAMES := bench-pthread bench-math bench-string string-benchset \
> + wcsmbs-benchset stdlib-benchset stdio-common-benchset math-benchset \
> + malloc-thread
> +INVALIDBENCHSETNAMES := $(filter-out ${VALIDBENCHSETNAMES},${BENCHSET})
> +ifneq (${INVALIDBENCHSETNAMES},)
> +$(info The following values in BENCHSET are invalid: ${INVALIDBENCHSETNAMES})
> +$(info The valid ones are: ${VALIDBENCHSETNAMES})
> +$(error Invalid BENCHSET value)
> +endif
> +endif
> +
> # Define the bench target only if the target has a usable python installation.
> ifdef PYTHON
> bench: bench-build bench-set bench-func bench-malloc
> @@ -170,10 +195,11 @@ bench-set: $(binaries-benchset)
> done
>
> bench-malloc: $(binaries-bench-malloc)
> - run=$(objpfx)bench-malloc-thread; \
> - for thr in 1 8 16 32; do \
> - echo "Running $${run} $${thr}"; \
> + for run in $^; do \
> + for thr in 1 8 16 32; do \
> + echo "Running $${run} $${thr}"; \
> $(run-bench) $${thr} > $${run}-$${thr}.out; \
> + done;\
> done
OK.
>
> # Build and execute the benchmark functions. This target generates JSON
> @@ -181,25 +207,28 @@ bench-malloc: $(binaries-bench-malloc)
> # so one could even execute them individually and process it using any JSON
> # capable language or tool.
> bench-func: $(binaries-bench)
> + if [ -n '$^' ] ; then \
> { timing_type=$$($(timing-type)); \
> - echo "{\"timing_type\": \"$${timing_type}\","; \
> - echo " \"functions\": {"; \
> - for run in $^; do \
> - if ! [ "x$${run}" = "x$<" ]; then \
> - echo ","; \
> + echo "{\"timing_type\": \"$${timing_type}\","; \
> + echo " \"functions\": {"; \
> + for run in $^; do \
> + if ! [ "x$${run}" = "x$<" ]; then \
> + echo ","; \
> + fi; \
> + echo "Running $${run}" >&2; \
> + $(run-bench) $(DETAILED_OPT); \
> + done; \
> + echo; \
> + echo " }"; \
> + echo "}"; \
> + } > $(objpfx)bench.out-tmp; \
> + if [ -f $(objpfx)bench.out ]; then \
> + mv -f $(objpfx)bench.out $(objpfx)bench.out.old; \
> fi; \
> - echo "Running $${run}" >&2; \
> - $(run-bench) $(DETAILED_OPT); \
> - done; \
> - echo; \
> - echo " }"; \
> - echo "}"; } > $(objpfx)bench.out-tmp; \
> - if [ -f $(objpfx)bench.out ]; then \
> - mv -f $(objpfx)bench.out $(objpfx)bench.out.old; \
> - fi; \
> - mv -f $(objpfx)bench.out-tmp $(objpfx)bench.out
> - $(PYTHON) scripts/validate_benchout.py $(objpfx)bench.out \
> - scripts/benchout.schema.json
> + mv -f $(objpfx)bench.out-tmp $(objpfx)bench.out; \
> + $(PYTHON) scripts/validate_benchout.py $(objpfx)bench.out \
> + scripts/benchout.schema.json; \
> + fi
>
> $(timing-type) $(binaries-bench) $(binaries-benchset) \
> $(binaries-bench-malloc): %: %.o $(objpfx)json-lib.o \
> diff --git a/benchtests/README b/benchtests/README
> index b015acf..20942c7 100644
> --- a/benchtests/README
> +++ b/benchtests/README
> @@ -53,6 +53,26 @@ otherwise the above command may try to build the benchmark again. Benchmarks
> that require generated code to be executed during the build are skipped when
> cross-building.
>
> +Running subsets of benchmarks:
> +==============================
> +
> +If a subset of benchmarks needs to be run one may run the benchmark by invoking
> +make as follows:
The following phrasing may be more succint:
"To run only a subset of benchmarks, one may invoke make as follows"
> +
> + $ make bench BENCHSET="bench-pthread bench-math malloc-thread"
> +
> +Allowing to measure the performance of specific glibc area. Posible areas are:
Drop this line and add:
"where BENCHSET may be a space-separated list of the following values:"
> +
> + bench-math
> + bench-pthread
> + bench-string
> + string-benchset
> + wcsmbs-benchset
> + stdlib-benchset
> + stdio-common-benchset
> + math-benchset
> + malloc-thread
> +
> Adding a function to benchtests:
> ===============================
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-11-22 10:34 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-16 22:54 [PATCH 1/3] benchtests: Wide range of tests names in schema.json Victor Rodriguez
2017-11-16 22:54 ` [PATCH 2/3] benchtests: Adjust valid and accepted properties Victor Rodriguez
2017-11-22 10:25 ` Siddhesh Poyarekar
2017-11-16 22:54 ` [PATCH 3/3] benchtests:Enable BENCHSET to run subset of tests Victor Rodriguez
2017-11-22 10:34 ` Siddhesh Poyarekar
2017-11-17 1:12 ` [PATCH 1/3] benchtests: Wide range of tests names in schema.json Paul Clarke
2017-11-18 14:24 ` Rodriguez Bahena, Victor
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).