[PATCH] build: All microbenchmarks should output in JSON. (bug 27740)

[PATCH] build: All microbenchmarks should output in JSON. (bug 27740)

[Girish Joshi]

[-- Attachment #1: Type: text/plain, Size: 630 bytes --]

Hello,
According to bug #27740 [1], I found that the following files need to
be updated.

    - benchtests/bench-memccpy.c
    - benchtests/bench-strcasestr.c
    - benchtests/bench-strcpy_chk.c
    - benchtests/bench-strsep.c
    - benchtests/bench-strtok.c

Also, the output json needs to be fixed in two files.
    - bench-math-inlines.c
    - bench-strcoll.c

This patch fixes both of these things.
The output is tested using
benchtests/scripts/compare_strings.py.

Please let me know if some correction is needed in this patch.

Thanks,
Girish Joshi
girishjoshi.io

[1]: https://sourceware.org/bugzilla/show_bug.cgi?id=27740

[-- Attachment #2: 0001-build-All-microbenchmarks-should-output-in-JSON.-bug.patch --]
[-- Type: text/x-patch, Size: 30167 bytes --]

From c8d1b68c8fc7cfb61d13f7321a7213cd1afb6d44 Mon Sep 17 00:00:00 2001
From: Girish Joshi <girish946@gmail.com>
Date: Sat, 15 Oct 2022 00:24:03 +0530
Subject: [PATCH] build: All microbenchmarks should output in JSON. (bug 27740)

This makes use of json-lib.h in
    - benchtests/bench-memccpy.c
    - benchtests/bench-strcasestr.c
    - benchtests/bench-strcpy_chk.c
    - benchtests/bench-strsep.c
    - benchtests/bench-strtok.c
to output to json.

This commit also fixes the json output generated by
bench-math-inlines.c and bench-strcoll.c

The output has been tested with
benchtests/scripts/compare_strings.py script.

Signed-off-by: Girish Joshi <girish946@gmail.com>
---
 benchtests/bench-math-inlines.c |   2 +
 benchtests/bench-memccpy.c      |  72 +++++++++++++------
 benchtests/bench-strcasestr.c   | 122 +++++++++++++++++++-------------
 benchtests/bench-strcoll.c      |   2 +
 benchtests/bench-strcpy_chk.c   | 107 +++++++++++++++++-----------
 benchtests/bench-strsep.c       | 122 ++++++++++++++++++++------------
 benchtests/bench-strtok.c       | 122 +++++++++++++++++++-------------
 7 files changed, 343 insertions(+), 206 deletions(-)

diff --git a/benchtests/bench-math-inlines.c b/benchtests/bench-math-inlines.c
index e755de1e28..cc49bcb869 100644
--- a/benchtests/bench-math-inlines.c
+++ b/benchtests/bench-math-inlines.c
@@ -248,6 +248,7 @@ test_main (void)
   bench_start ();
 
   json_init (&json_ctx, 2, stdout);
+  json_document_begin (&json_ctx);
   json_attr_object_begin (&json_ctx, TEST_NAME);
 
   /* Create 2 test arrays, one with 10% zeroes, 10% negative values,
@@ -278,6 +279,7 @@ test_main (void)
     }
 
   json_attr_object_end (&json_ctx);
+  json_document_end (&json_ctx);
   return 0;
 }
 
diff --git a/benchtests/bench-memccpy.c b/benchtests/bench-memccpy.c
index 51ed8dad2f..5302cacd0b 100644
--- a/benchtests/bench-memccpy.c
+++ b/benchtests/bench-memccpy.c
@@ -35,11 +35,13 @@ generic_memccpy (void *dst, const void *src, int c, size_t n)
 IMPL (memccpy, 1)
 IMPL (generic_memccpy, 0)
 
+#include "json-lib.h"
+
 typedef void *(*proto_t) (void *, const void *, int c, size_t);
 
 static void
-do_one_test (impl_t *impl, void *dst, const void *src, int c, size_t len,
-	     size_t n)
+do_one_test (json_ctx_t *json_ctx, impl_t *impl, void *dst, const void *src,
+   int c, size_t len, size_t n)
 {
   size_t i, iters = INNER_LOOP_ITERS_LARGE;
   timing_t start, stop, cur;
@@ -53,12 +55,12 @@ do_one_test (impl_t *impl, void *dst, const void *src, int c, size_t len,
 
   TIMING_DIFF (cur, start, stop);
 
-  TIMING_PRINT_MEAN ((double) cur, (double) iters);
+  json_element_double (json_ctx, (double) cur / (double) iters);
 }
 
 static void
-do_test (size_t align1, size_t align2, int c, size_t len, size_t n,
-	 int max_char)
+do_test (json_ctx_t *json_ctx, size_t align1, size_t align2, int c,
+   size_t len, size_t n, int max_char)
 {
   size_t i;
   char *s1, *s2;
@@ -84,46 +86,70 @@ do_test (size_t align1, size_t align2, int c, size_t len, size_t n,
   for (i = len; i + align1 < page_size && i < len + 64; ++i)
     s1[i] = 32 + 32 * i % (max_char - 32);
 
-  printf ("Length %4zd, n %4zd, char %d, alignment %2zd/%2zd:", len, n, c, align1, align2);
+  json_element_object_begin (json_ctx);
+  json_attr_uint (json_ctx, "length", (double) len);
+  json_attr_uint (json_ctx, "n", (double) n);
+  json_attr_uint (json_ctx, "c", (int) c);
+  json_attr_uint (json_ctx, "align1", (double) align1);
+  json_attr_uint (json_ctx, "align2", (double) align2);
+  json_array_begin (json_ctx, "timings");
 
   FOR_EACH_IMPL (impl, 0)
-    do_one_test (impl, s2, s1, c, len, n);
+    do_one_test (json_ctx, impl, s2, s1, c, len, n);
 
-  putchar ('\n');
+  json_array_end (json_ctx);
+  json_element_object_end (json_ctx);
 }
 
 int
 test_main (void)
 {
+  json_ctx_t json_ctx;
   size_t i;
 
   test_init ();
 
-  printf ("%28s", "");
+  json_init (&json_ctx, 0, stdout);
+
+  json_document_begin (&json_ctx);
+  json_attr_string (&json_ctx, "timing_type", TIMING_TYPE);
+
+  json_attr_object_begin (&json_ctx, "functions");
+  json_attr_object_begin (&json_ctx, TEST_NAME);
+  json_attr_string (&json_ctx, "bench-variant", "default");
+
+  json_array_begin (&json_ctx, "ifuncs");
+
   FOR_EACH_IMPL (impl, 0)
-    printf ("\t%s", impl->name);
-  putchar ('\n');
+    json_element_string (&json_ctx, impl->name);
+  json_array_end (&json_ctx);
 
+  json_array_begin (&json_ctx, "results");
   for (i = 1; i < 8; ++i)
     {
-      do_test (i, i, 12, 16, 16, 127);
-      do_test (i, i, 23, 16, 16, 255);
-      do_test (i, 2 * i, 28, 16, 16, 127);
-      do_test (2 * i, i, 31, 16, 16, 255);
-      do_test (8 - i, 2 * i, 1, 1 << i, 2 << i, 127);
-      do_test (2 * i, 8 - i, 17, 2 << i, 1 << i, 127);
-      do_test (8 - i, 2 * i, 0, 1 << i, 2 << i, 255);
-      do_test (2 * i, 8 - i, i, 2 << i, 1 << i, 255);
+      do_test (&json_ctx, i, i, 12, 16, 16, 127);
+      do_test (&json_ctx, i, i, 23, 16, 16, 255);
+      do_test (&json_ctx, i, 2 * i, 28, 16, 16, 127);
+      do_test (&json_ctx, 2 * i, i, 31, 16, 16, 255);
+      do_test (&json_ctx, 8 - i, 2 * i, 1, 1 << i, 2 << i, 127);
+      do_test (&json_ctx, 2 * i, 8 - i, 17, 2 << i, 1 << i, 127);
+      do_test (&json_ctx, 8 - i, 2 * i, 0, 1 << i, 2 << i, 255);
+      do_test (&json_ctx, 2 * i, 8 - i, i, 2 << i, 1 << i, 255);
     }
 
   for (i = 1; i < 8; ++i)
     {
-      do_test (0, 0, i, 4 << i, 8 << i, 127);
-      do_test (0, 0, i, 16 << i, 8 << i, 127);
-      do_test (8 - i, 2 * i, i, 4 << i, 8 << i, 127);
-      do_test (8 - i, 2 * i, i, 16 << i, 8 << i, 127);
+      do_test (&json_ctx, 0, 0, i, 4 << i, 8 << i, 127);
+      do_test (&json_ctx, 0, 0, i, 16 << i, 8 << i, 127);
+      do_test (&json_ctx, 8 - i, 2 * i, i, 4 << i, 8 << i, 127);
+      do_test (&json_ctx, 8 - i, 2 * i, i, 16 << i, 8 << i, 127);
     }
 
+  json_array_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_document_end (&json_ctx);
+
   return ret;
 }
 
diff --git a/benchtests/bench-strcasestr.c b/benchtests/bench-strcasestr.c
index b3b26406f9..a8bcfb3aec 100644
--- a/benchtests/bench-strcasestr.c
+++ b/benchtests/bench-strcasestr.c
@@ -32,9 +32,11 @@ typedef char *(*proto_t) (const char *, const char *);
 IMPL (simple_strcasestr, 0)
 IMPL (strcasestr, 1)
 
+# include "json-lib.h"
 
 static void
-do_one_test (impl_t *impl, const char *s1, const char *s2, char *exp_result)
+do_one_test (json_ctx_t *json_ctx, impl_t *impl, const char *s1,
+   const char *s2, char *exp_result)
 {
   size_t i, iters = INNER_LOOP_ITERS_SMALL;
   timing_t start, stop, cur;
@@ -48,13 +50,13 @@ do_one_test (impl_t *impl, const char *s1, const char *s2, char *exp_result)
 
   TIMING_DIFF (cur, start, stop);
 
-  TIMING_PRINT_MEAN ((double) cur, (double) iters);
+  json_element_double (json_ctx, (double) cur / (double) iters);
 }
 
 
 static void
-do_test (size_t align1, size_t align2, size_t len1, size_t len2,
-	 int fail)
+do_test (json_ctx_t *json_ctx, size_t align1, size_t align2, size_t len1,
+   size_t len2, int fail)
 {
   char *s1 = (char *) (buf1 + align1);
   char *s2 = (char *) (buf2 + align2);
@@ -88,67 +90,91 @@ do_test (size_t align1, size_t align2, size_t len1, size_t len2,
     }
   s1[len1] = '\0';
 
-  printf ("Length %4zd/%zd, alignment %2zd/%2zd, %s:",
-	  len1, len2, align1, align2, fail ? "fail" : "found");
+  json_element_object_begin (json_ctx);
+  json_attr_uint (json_ctx, "length1", (double) len1);
+  json_attr_uint (json_ctx, "length2", (double) len2);
+  json_attr_uint (json_ctx, "align1", (double) align1);
+  json_attr_uint (json_ctx, "align2", (double) align2);
+  json_attr_string (json_ctx, "fail", fail ? "fail" : "found");
+  json_array_begin (json_ctx, "timings");
 
   FOR_EACH_IMPL (impl, 0)
-    do_one_test (impl, s1, s2, fail ? NULL : s1 + len1 - len2);
+    do_one_test (json_ctx, impl, s1, s2, fail ? NULL : s1 + len1 - len2);
 
-  putchar ('\n');
+  json_array_end (json_ctx);
+  json_element_object_end (json_ctx);
 }
 
 static int
 test_main (void)
 {
+  json_ctx_t json_ctx;
+
   test_init ();
 
-  printf ("%23s", "");
+  json_init (&json_ctx, 0, stdout);
+
+  json_document_begin (&json_ctx);
+  json_attr_string (&json_ctx, "timing_type", TIMING_TYPE);
+
+  json_attr_object_begin (&json_ctx, "functions");
+  json_attr_object_begin (&json_ctx, TEST_NAME);
+  json_attr_string (&json_ctx, "bench-variant", "default");
+
+  json_array_begin (&json_ctx, "ifuncs");
+
   FOR_EACH_IMPL (impl, 0)
-    printf ("\t%s", impl->name);
-  putchar ('\n');
+    json_element_string(&json_ctx, impl->name); 
+  json_array_end (&json_ctx);
 
+  json_array_begin (&json_ctx, "results");
   for (size_t klen = 2; klen < 32; ++klen)
     for (size_t hlen = 2 * klen; hlen < 16 * klen; hlen += klen)
       {
-	do_test (0, 0, hlen, klen, 0);
-	do_test (0, 0, hlen, klen, 1);
-	do_test (0, 3, hlen, klen, 0);
-	do_test (0, 3, hlen, klen, 1);
-	do_test (0, 9, hlen, klen, 0);
-	do_test (0, 9, hlen, klen, 1);
-	do_test (0, 15, hlen, klen, 0);
-	do_test (0, 15, hlen, klen, 1);
-
-	do_test (3, 0, hlen, klen, 0);
-	do_test (3, 0, hlen, klen, 1);
-	do_test (3, 3, hlen, klen, 0);
-	do_test (3, 3, hlen, klen, 1);
-	do_test (3, 9, hlen, klen, 0);
-	do_test (3, 9, hlen, klen, 1);
-	do_test (3, 15, hlen, klen, 0);
-	do_test (3, 15, hlen, klen, 1);
-
-	do_test (9, 0, hlen, klen, 0);
-	do_test (9, 0, hlen, klen, 1);
-	do_test (9, 3, hlen, klen, 0);
-	do_test (9, 3, hlen, klen, 1);
-	do_test (9, 9, hlen, klen, 0);
-	do_test (9, 9, hlen, klen, 1);
-	do_test (9, 15, hlen, klen, 0);
-	do_test (9, 15, hlen, klen, 1);
-
-	do_test (15, 0, hlen, klen, 0);
-	do_test (15, 0, hlen, klen, 1);
-	do_test (15, 3, hlen, klen, 0);
-	do_test (15, 3, hlen, klen, 1);
-	do_test (15, 9, hlen, klen, 0);
-	do_test (15, 9, hlen, klen, 1);
-	do_test (15, 15, hlen, klen, 0);
-	do_test (15, 15, hlen, klen, 1);
+	do_test (&json_ctx, 0, 0, hlen, klen, 0);
+	do_test (&json_ctx, 0, 0, hlen, klen, 1);
+	do_test (&json_ctx, 0, 3, hlen, klen, 0);
+	do_test (&json_ctx, 0, 3, hlen, klen, 1);
+	do_test (&json_ctx, 0, 9, hlen, klen, 0);
+	do_test (&json_ctx, 0, 9, hlen, klen, 1);
+	do_test (&json_ctx, 0, 15, hlen, klen, 0);
+	do_test (&json_ctx, 0, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 3, 0, hlen, klen, 0);
+	do_test (&json_ctx, 3, 0, hlen, klen, 1);
+	do_test (&json_ctx, 3, 3, hlen, klen, 0);
+	do_test (&json_ctx, 3, 3, hlen, klen, 1);
+	do_test (&json_ctx, 3, 9, hlen, klen, 0);
+	do_test (&json_ctx, 3, 9, hlen, klen, 1);
+	do_test (&json_ctx, 3, 15, hlen, klen, 0);
+	do_test (&json_ctx, 3, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 9, 0, hlen, klen, 0);
+	do_test (&json_ctx, 9, 0, hlen, klen, 1);
+	do_test (&json_ctx, 9, 3, hlen, klen, 0);
+	do_test (&json_ctx, 9, 3, hlen, klen, 1);
+	do_test (&json_ctx, 9, 9, hlen, klen, 0);
+	do_test (&json_ctx, 9, 9, hlen, klen, 1);
+	do_test (&json_ctx, 9, 15, hlen, klen, 0);
+	do_test (&json_ctx, 9, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 15, 0, hlen, klen, 0);
+	do_test (&json_ctx, 15, 0, hlen, klen, 1);
+	do_test (&json_ctx, 15, 3, hlen, klen, 0);
+	do_test (&json_ctx, 15, 3, hlen, klen, 1);
+	do_test (&json_ctx, 15, 9, hlen, klen, 0);
+	do_test (&json_ctx, 15, 9, hlen, klen, 1);
+	do_test (&json_ctx, 15, 15, hlen, klen, 0);
+	do_test (&json_ctx, 15, 15, hlen, klen, 1);
       }
 
-  do_test (0, 0, page_size - 1, 16, 0);
-  do_test (0, 0, page_size - 1, 16, 1);
+  do_test (&json_ctx, 0, 0, page_size - 1, 16, 0);
+  do_test (&json_ctx, 0, 0, page_size - 1, 16, 1);
+
+  json_array_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_document_end (&json_ctx);
 
   return ret;
 }
diff --git a/benchtests/bench-strcoll.c b/benchtests/bench-strcoll.c
index b3cd5ce8a9..08f81dea9e 100644
--- a/benchtests/bench-strcoll.c
+++ b/benchtests/bench-strcoll.c
@@ -239,6 +239,7 @@ main (void)
   json_ctx_t *json_ctx = malloc (sizeof (json_ctx_t));
   assert (json_ctx != NULL);
   json_init (json_ctx, 2, stdout);
+  json_document_begin (json_ctx);
   json_attr_object_begin (json_ctx, "strcoll");
 
   size_t i;
@@ -271,6 +272,7 @@ main (void)
 
 out:
   json_attr_object_end (json_ctx);
+  json_document_end (json_ctx);
   free (json_ctx);
   return result;
 }
diff --git a/benchtests/bench-strcpy_chk.c b/benchtests/bench-strcpy_chk.c
index 6fce90cc98..90842fac52 100644
--- a/benchtests/bench-strcpy_chk.c
+++ b/benchtests/bench-strcpy_chk.c
@@ -54,6 +54,7 @@ simple_strcpy_chk (char *dst, const char *src, size_t len)
 #include <signal.h>
 
 #include <support/support.h>
+#include "json-lib.h"
 
 volatile int chk_fail_ok;
 jmp_buf chk_fail_buf;
@@ -73,7 +74,7 @@ handler (int sig)
 typedef char *(*proto_t) (char *, const char *, size_t);
 
 static void
-do_one_test (impl_t *impl, char *dst, const char *src,
+do_one_test (json_ctx_t *json_ctx, impl_t *impl, char *dst, const char *src,
 	     size_t len, size_t dlen)
 {
   char *res;
@@ -124,11 +125,12 @@ do_one_test (impl_t *impl, char *dst, const char *src,
 
   TIMING_DIFF (cur, start, stop);
 
-  TIMING_PRINT_MEAN ((double) cur, (double) iters);
+  json_element_double (json_ctx, (double) cur / (double) iters);
 }
 
 static void
-do_test (size_t align1, size_t align2, size_t len, size_t dlen, int max_char)
+do_test (json_ctx_t *json_ctx, size_t align1, size_t align2, size_t len,
+	 size_t dlen, int max_char)
 {
   size_t i;
   char *s1, *s2;
@@ -148,96 +150,119 @@ do_test (size_t align1, size_t align2, size_t len, size_t dlen, int max_char)
     s1[i] = 32 + 23 * i % (max_char - 32);
   s1[len] = 0;
 
-  if (dlen > len)
-    printf ("Length %4zd, alignment %2zd/%2zd:", len, align1, align2);
+  if (dlen > len){
+    json_element_object_begin (json_ctx);
+    json_attr_uint (json_ctx, "length", (double) len);
+    json_attr_uint (json_ctx, "align1", (double) align1);
+    json_attr_uint (json_ctx, "align2", (double) align2);
+    json_array_begin (json_ctx, "timings");
+  }
 
   FOR_EACH_IMPL (impl, 0)
-    do_one_test (impl, s2, s1, len, dlen);
+    do_one_test (json_ctx, impl, s2, s1, len, dlen);
 
-  if (dlen > len)
-    putchar ('\n');
+  if (dlen > len){
+    json_array_end (json_ctx);
+    json_element_object_end (json_ctx);
+  }
 }
 
 static int
 test_main (void)
 {
+  json_ctx_t json_ctx;
   size_t i;
 
   set_fortify_handler (handler);
 
   test_init ();
+  json_init (&json_ctx, 0, stdout);
+
+  json_document_begin (&json_ctx);
+  json_attr_string (&json_ctx, "timing_type", TIMING_TYPE);
+
+  json_attr_object_begin (&json_ctx, "functions");
+  json_attr_object_begin (&json_ctx, TEST_NAME);
+  json_attr_string (&json_ctx, "bench-variant", "default");
+
+  json_array_begin (&json_ctx, "ifuncs");
 
-  printf ("%23s", "");
   FOR_EACH_IMPL (impl, 0)
-    printf ("\t%s", impl->name);
-  putchar ('\n');
+    json_element_string (&json_ctx, impl->name);
+  json_array_end (&json_ctx);
 
+  json_array_begin (&json_ctx, "results");
   for (i = 0; i < 16; ++i)
     {
-      do_test (0, 0, i, i + 1, 127);
-      do_test (0, 0, i, i + 1, 255);
-      do_test (0, i, i, i + 1, 127);
-      do_test (i, 0, i, i + 1, 255);
+      do_test (&json_ctx, 0, 0, i, i + 1, 127);
+      do_test (&json_ctx, 0, 0, i, i + 1, 255);
+      do_test (&json_ctx, 0, i, i, i + 1, 127);
+      do_test (&json_ctx, i, 0, i, i + 1, 255);
     }
 
   for (i = 1; i < 8; ++i)
     {
-      do_test (0, 0, 8 << i, (8 << i) + 1, 127);
-      do_test (8 - i, 2 * i, (8 << i), (8 << i) + 1, 127);
+      do_test (&json_ctx, 0, 0, 8 << i, (8 << i) + 1, 127);
+      do_test (&json_ctx, 8 - i, 2 * i, (8 << i), (8 << i) + 1, 127);
     }
 
   for (i = 1; i < 8; ++i)
     {
-      do_test (i, 2 * i, (8 << i), (8 << i) + 1, 127);
-      do_test (2 * i, i, (8 << i), (8 << i) + 1, 255);
-      do_test (i, i, (8 << i), (8 << i) + 1, 127);
-      do_test (i, i, (8 << i), (8 << i) + 1, 255);
+      do_test (&json_ctx, i, 2 * i, (8 << i), (8 << i) + 1, 127);
+      do_test (&json_ctx, 2 * i, i, (8 << i), (8 << i) + 1, 255);
+      do_test (&json_ctx, i, i, (8 << i), (8 << i) + 1, 127);
+      do_test (&json_ctx, i, i, (8 << i), (8 << i) + 1, 255);
     }
 
   for (i = 0; i < 16; ++i)
     {
-      do_test (0, 0, i, i + 256, 127);
-      do_test (0, 0, i, i + 256, 255);
-      do_test (0, i, i, i + 256, 127);
-      do_test (i, 0, i, i + 256, 255);
+      do_test (&json_ctx, 0, 0, i, i + 256, 127);
+      do_test (&json_ctx, 0, 0, i, i + 256, 255);
+      do_test (&json_ctx, 0, i, i, i + 256, 127);
+      do_test (&json_ctx, i, 0, i, i + 256, 255);
     }
 
   for (i = 1; i < 8; ++i)
     {
-      do_test (0, 0, 8 << i, (8 << i) + 256, 127);
-      do_test (8 - i, 2 * i, (8 << i), (8 << i) + 256, 127);
+      do_test (&json_ctx, 0, 0, 8 << i, (8 << i) + 256, 127);
+      do_test (&json_ctx, 8 - i, 2 * i, (8 << i), (8 << i) + 256, 127);
     }
 
   for (i = 1; i < 8; ++i)
     {
-      do_test (i, 2 * i, (8 << i), (8 << i) + 256, 127);
-      do_test (2 * i, i, (8 << i), (8 << i) + 256, 255);
-      do_test (i, i, (8 << i), (8 << i) + 256, 127);
-      do_test (i, i, (8 << i), (8 << i) + 256, 255);
+      do_test (&json_ctx, i, 2 * i, (8 << i), (8 << i) + 256, 127);
+      do_test (&json_ctx, 2 * i, i, (8 << i), (8 << i) + 256, 255);
+      do_test (&json_ctx, i, i, (8 << i), (8 << i) + 256, 127);
+      do_test (&json_ctx, i, i, (8 << i), (8 << i) + 256, 255);
     }
 
   for (i = 0; i < 16; ++i)
     {
-      do_test (0, 0, i, i, 127);
-      do_test (0, 0, i, i + 2, 255);
-      do_test (0, i, i, i + 3, 127);
-      do_test (i, 0, i, i + 4, 255);
+      do_test (&json_ctx, 0, 0, i, i, 127);
+      do_test (&json_ctx, 0, 0, i, i + 2, 255);
+      do_test (&json_ctx, 0, i, i, i + 3, 127);
+      do_test (&json_ctx, i, 0, i, i + 4, 255);
     }
 
   for (i = 1; i < 8; ++i)
     {
-      do_test (0, 0, 8 << i, (8 << i) - 15, 127);
-      do_test (8 - i, 2 * i, (8 << i), (8 << i) + 5, 127);
+      do_test (&json_ctx, 0, 0, 8 << i, (8 << i) - 15, 127);
+      do_test (&json_ctx, 8 - i, 2 * i, (8 << i), (8 << i) + 5, 127);
     }
 
   for (i = 1; i < 8; ++i)
     {
-      do_test (i, 2 * i, (8 << i), (8 << i) + i, 127);
-      do_test (2 * i, i, (8 << i), (8 << i) + (i - 1), 255);
-      do_test (i, i, (8 << i), (8 << i) + i + 2, 127);
-      do_test (i, i, (8 << i), (8 << i) + i + 3, 255);
+      do_test (&json_ctx, i, 2 * i, (8 << i), (8 << i) + i, 127);
+      do_test (&json_ctx, 2 * i, i, (8 << i), (8 << i) + (i - 1), 255);
+      do_test (&json_ctx, i, i, (8 << i), (8 << i) + i + 2, 127);
+      do_test (&json_ctx, i, i, (8 << i), (8 << i) + i + 3, 255);
     }
 
+  json_array_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_document_end (&json_ctx);
+
   return 0;
 }
 
diff --git a/benchtests/bench-strsep.c b/benchtests/bench-strsep.c
index 4b203aff30..708689b498 100644
--- a/benchtests/bench-strsep.c
+++ b/benchtests/bench-strsep.c
@@ -20,6 +20,8 @@
 #define TEST_NAME "strsep"
 #include "bench-string.h"
 
+# include "json-lib.h"
+
 char *
 simple_strsep (char **s1, char *s2)
 {
@@ -101,7 +103,8 @@ IMPL (strsep, 1)
 IMPL (oldstrsep, 2)
 
 static void
-do_one_test (impl_t * impl, const char *s1, const char *s2)
+do_one_test (json_ctx_t *json_ctx, impl_t * impl, const char *s1,
+   const char *s2)
 {
   size_t i, iters = INNER_LOOP_ITERS_SMALL;
   timing_t start, stop, cur;
@@ -118,11 +121,12 @@ do_one_test (impl_t * impl, const char *s1, const char *s2)
 
   TIMING_DIFF (cur, start, stop);
 
-  TIMING_PRINT_MEAN ((double) cur, (double) iters);
+  json_element_double (json_ctx, (double) cur / (double) iters);
 }
 
 static void
-do_test (size_t align1, size_t align2, size_t len1, size_t len2, int fail)
+do_test (json_ctx_t *json_ctx, size_t align1, size_t align2, size_t len1,
+   size_t len2, int fail)
 {
   char *s2 = (char *) (buf2 + align2);
 
@@ -138,8 +142,13 @@ do_test (size_t align1, size_t align2, size_t len1, size_t len2, int fail)
     }
   s2[len2] = '\0';
 
-  printf ("Length %4zd/%zd, alignment %2zd/%2zd, %s:",
-	  len1, len2, align1, align2, fail ? "fail" : "found");
+  json_element_object_begin (json_ctx);
+  json_attr_uint (json_ctx, "length1", (double) len1);
+  json_attr_uint (json_ctx, "length2", (double) len2);
+  json_attr_uint (json_ctx, "align1", (double) align1);
+  json_attr_uint (json_ctx, "align2", (double) align2);
+  json_attr_string  (json_ctx, "fail", fail ? "fail" : "found");
+  json_array_begin (json_ctx, "timings");
 
   FOR_EACH_IMPL (impl, 0)
   {
@@ -148,62 +157,83 @@ do_test (size_t align1, size_t align2, size_t len1, size_t len2, int fail)
     if (!fail)
       s1[len1 / 2] = '1';
     s1[len1] = '\0';
-    do_one_test (impl, s1, s2);
+    do_one_test (json_ctx, impl, s1, s2);
   }
-  putchar ('\n');
+
+  json_array_end (json_ctx);
+  json_element_object_end (json_ctx);
 }
 
 static int
 test_main (void)
 {
+  json_ctx_t json_ctx;
+
   test_init ();
 
+  json_init (&json_ctx, 0, stdout);
+
+  json_document_begin (&json_ctx);
+  json_attr_string (&json_ctx, "timing_type", TIMING_TYPE);
+
+  json_attr_object_begin (&json_ctx, "functions");
+  json_attr_object_begin (&json_ctx, TEST_NAME);
+  json_attr_string (&json_ctx, "bench-variant", "default");
+
+  json_array_begin (&json_ctx, "ifuncs");
+
   printf ("%23s", "");
   FOR_EACH_IMPL (impl, 0)
-    printf ("\t%s", impl->name);
-  putchar ('\n');
+    json_element_string (&json_ctx, impl->name);
+  json_array_end (&json_ctx);
 
+  json_array_begin (&json_ctx, "results");
   for (size_t klen = 2; klen < 32; ++klen)
     for (size_t hlen = 4 * klen; hlen < 8 * klen; hlen += klen)
       {
-	do_test (0, 0, hlen, klen, 0);
-	do_test (0, 0, hlen, klen, 1);
-	do_test (0, 3, hlen, klen, 0);
-	do_test (0, 3, hlen, klen, 1);
-	do_test (0, 9, hlen, klen, 0);
-	do_test (0, 9, hlen, klen, 1);
-	do_test (0, 15, hlen, klen, 0);
-	do_test (0, 15, hlen, klen, 1);
-
-	do_test (3, 0, hlen, klen, 0);
-	do_test (3, 0, hlen, klen, 1);
-	do_test (3, 3, hlen, klen, 0);
-	do_test (3, 3, hlen, klen, 1);
-	do_test (3, 9, hlen, klen, 0);
-	do_test (3, 9, hlen, klen, 1);
-	do_test (3, 15, hlen, klen, 0);
-	do_test (3, 15, hlen, klen, 1);
-
-	do_test (9, 0, hlen, klen, 0);
-	do_test (9, 0, hlen, klen, 1);
-	do_test (9, 3, hlen, klen, 0);
-	do_test (9, 3, hlen, klen, 1);
-	do_test (9, 9, hlen, klen, 0);
-	do_test (9, 9, hlen, klen, 1);
-	do_test (9, 15, hlen, klen, 0);
-	do_test (9, 15, hlen, klen, 1);
-
-	do_test (15, 0, hlen, klen, 0);
-	do_test (15, 0, hlen, klen, 1);
-	do_test (15, 3, hlen, klen, 0);
-	do_test (15, 3, hlen, klen, 1);
-	do_test (15, 9, hlen, klen, 0);
-	do_test (15, 9, hlen, klen, 1);
-	do_test (15, 15, hlen, klen, 0);
-	do_test (15, 15, hlen, klen, 1);
+	do_test (&json_ctx, 0, 0, hlen, klen, 0);
+	do_test (&json_ctx, 0, 0, hlen, klen, 1);
+	do_test (&json_ctx, 0, 3, hlen, klen, 0);
+	do_test (&json_ctx, 0, 3, hlen, klen, 1);
+	do_test (&json_ctx, 0, 9, hlen, klen, 0);
+	do_test (&json_ctx, 0, 9, hlen, klen, 1);
+	do_test (&json_ctx, 0, 15, hlen, klen, 0);
+	do_test (&json_ctx, 0, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 3, 0, hlen, klen, 0);
+	do_test (&json_ctx, 3, 0, hlen, klen, 1);
+	do_test (&json_ctx, 3, 3, hlen, klen, 0);
+	do_test (&json_ctx, 3, 3, hlen, klen, 1);
+	do_test (&json_ctx, 3, 9, hlen, klen, 0);
+	do_test (&json_ctx, 3, 9, hlen, klen, 1);
+	do_test (&json_ctx, 3, 15, hlen, klen, 0);
+	do_test (&json_ctx, 3, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 9, 0, hlen, klen, 0);
+	do_test (&json_ctx, 9, 0, hlen, klen, 1);
+	do_test (&json_ctx, 9, 3, hlen, klen, 0);
+	do_test (&json_ctx, 9, 3, hlen, klen, 1);
+	do_test (&json_ctx, 9, 9, hlen, klen, 0);
+	do_test (&json_ctx, 9, 9, hlen, klen, 1);
+	do_test (&json_ctx, 9, 15, hlen, klen, 0);
+	do_test (&json_ctx, 9, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 15, 0, hlen, klen, 0);
+	do_test (&json_ctx, 15, 0, hlen, klen, 1);
+	do_test (&json_ctx, 15, 3, hlen, klen, 0);
+	do_test (&json_ctx, 15, 3, hlen, klen, 1);
+	do_test (&json_ctx, 15, 9, hlen, klen, 0);
+	do_test (&json_ctx, 15, 9, hlen, klen, 1);
+	do_test (&json_ctx, 15, 15, hlen, klen, 0);
+	do_test (&json_ctx, 15, 15, hlen, klen, 1);
       }
-  do_test (0, 0, page_size - 1, 16, 0);
-  do_test (0, 0, page_size - 1, 16, 1);
+  do_test (&json_ctx, 0, 0, page_size - 1, 16, 0);
+  do_test (&json_ctx, 0, 0, page_size - 1, 16, 1);
+
+  json_array_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_document_end (&json_ctx);
 
   return ret;
 }
diff --git a/benchtests/bench-strtok.c b/benchtests/bench-strtok.c
index 711bdaab58..4a4df16f41 100644
--- a/benchtests/bench-strtok.c
+++ b/benchtests/bench-strtok.c
@@ -57,8 +57,11 @@ typedef char *(*proto_t) (const char *, const char *);
 IMPL (oldstrtok, 0)
 IMPL (strtok, 1)
 
+# include "json-lib.h"
+
 static void
-do_one_test (impl_t * impl, const char *s1, const char *s2)
+do_one_test (json_ctx_t *json_ctx, impl_t * impl, const char *s1,
+   const char *s2)
 {
   size_t i, iters = INNER_LOOP_ITERS_SMALL;
   timing_t start, stop, cur;
@@ -73,13 +76,13 @@ do_one_test (impl_t * impl, const char *s1, const char *s2)
 
   TIMING_DIFF (cur, start, stop);
 
-  TIMING_PRINT_MEAN ((double) cur, (double) iters);
-
+  json_element_double (json_ctx, (double) cur / (double) iters);
 }
 
 
 static void
-do_test (size_t align1, size_t align2, size_t len1, size_t len2, int fail)
+do_test (json_ctx_t *json_ctx, size_t align1, size_t align2, size_t len1,
+   size_t len2, int fail)
 {
   char *s2 = (char *) (buf2 + align2);
   static const char d[] = "1234567890abcdef";
@@ -92,8 +95,13 @@ do_test (size_t align1, size_t align2, size_t len1, size_t len2, int fail)
     }
   s2[len2] = '\0';
 
-  printf ("Length %4zd/%zd, alignment %2zd/%2zd, %s:",
-	  len1, len2, align1, align2, fail ? "fail" : "found");
+  json_element_object_begin (json_ctx);
+  json_attr_uint (json_ctx, "length1", (double) len1);
+  json_attr_uint (json_ctx, "length2", (double) len2);
+  json_attr_uint (json_ctx, "align1", (double) align1);
+  json_attr_uint (json_ctx, "align2", (double) align2);
+  json_attr_string (json_ctx, "fail", fail ? "fail" : "found");
+  json_array_begin (json_ctx, "timings");
 
   FOR_EACH_IMPL (impl, 0)
   {
@@ -117,62 +125,80 @@ do_test (size_t align1, size_t align2, size_t len1, size_t len2, int fail)
 	  memcpy (s1 + (len1 - len2) - (3 * len2), s2, len2);
       }
     s1[len1] = '\0';
-    do_one_test (impl, s1, s2);
+    do_one_test (json_ctx, impl, s1, s2);
   }
-  putchar ('\n');
+
+  json_array_end (json_ctx);
+  json_element_object_end (json_ctx);
 }
 
 static int
 test_main (void)
 {
+  json_ctx_t json_ctx;
   test_init ();
 
-  printf ("%23s", "");
+  json_init (&json_ctx, 0, stdout);
+
+  json_document_begin (&json_ctx);
+  json_attr_string (&json_ctx, "timing_type", TIMING_TYPE);
+
+  json_attr_object_begin (&json_ctx, "functions");
+  json_attr_object_begin (&json_ctx, TEST_NAME);
+  json_attr_string (&json_ctx, "bench-variant", "default");
+
+  json_array_begin (&json_ctx, "ifuncs");
   FOR_EACH_IMPL (impl, 0)
-    printf ("\t%s", impl->name);
-  putchar ('\n');
+    json_element_string (&json_ctx, impl->name);
+  json_array_end (&json_ctx);
 
+  json_array_begin (&json_ctx, "results");
   for (size_t klen = 2; klen < 32; ++klen)
     for (size_t hlen = 2 * klen; hlen < 16 * klen; hlen += klen)
       {
-	do_test (0, 0, hlen, klen, 0);
-	do_test (0, 0, hlen, klen, 1);
-	do_test (0, 3, hlen, klen, 0);
-	do_test (0, 3, hlen, klen, 1);
-	do_test (0, 9, hlen, klen, 0);
-	do_test (0, 9, hlen, klen, 1);
-	do_test (0, 15, hlen, klen, 0);
-	do_test (0, 15, hlen, klen, 1);
-
-	do_test (3, 0, hlen, klen, 0);
-	do_test (3, 0, hlen, klen, 1);
-	do_test (3, 3, hlen, klen, 0);
-	do_test (3, 3, hlen, klen, 1);
-	do_test (3, 9, hlen, klen, 0);
-	do_test (3, 9, hlen, klen, 1);
-	do_test (3, 15, hlen, klen, 0);
-	do_test (3, 15, hlen, klen, 1);
-
-	do_test (9, 0, hlen, klen, 0);
-	do_test (9, 0, hlen, klen, 1);
-	do_test (9, 3, hlen, klen, 0);
-	do_test (9, 3, hlen, klen, 1);
-	do_test (9, 9, hlen, klen, 0);
-	do_test (9, 9, hlen, klen, 1);
-	do_test (9, 15, hlen, klen, 0);
-	do_test (9, 15, hlen, klen, 1);
-
-	do_test (15, 0, hlen, klen, 0);
-	do_test (15, 0, hlen, klen, 1);
-	do_test (15, 3, hlen, klen, 0);
-	do_test (15, 3, hlen, klen, 1);
-	do_test (15, 9, hlen, klen, 0);
-	do_test (15, 9, hlen, klen, 1);
-	do_test (15, 15, hlen, klen, 0);
-	do_test (15, 15, hlen, klen, 1);
+	do_test (&json_ctx, 0, 0, hlen, klen, 0);
+	do_test (&json_ctx, 0, 0, hlen, klen, 1);
+	do_test (&json_ctx, 0, 3, hlen, klen, 0);
+	do_test (&json_ctx, 0, 3, hlen, klen, 1);
+	do_test (&json_ctx, 0, 9, hlen, klen, 0);
+	do_test (&json_ctx, 0, 9, hlen, klen, 1);
+	do_test (&json_ctx, 0, 15, hlen, klen, 0);
+	do_test (&json_ctx, 0, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 3, 0, hlen, klen, 0);
+	do_test (&json_ctx, 3, 0, hlen, klen, 1);
+	do_test (&json_ctx, 3, 3, hlen, klen, 0);
+	do_test (&json_ctx, 3, 3, hlen, klen, 1);
+	do_test (&json_ctx, 3, 9, hlen, klen, 0);
+	do_test (&json_ctx, 3, 9, hlen, klen, 1);
+	do_test (&json_ctx, 3, 15, hlen, klen, 0);
+	do_test (&json_ctx, 3, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 9, 0, hlen, klen, 0);
+	do_test (&json_ctx, 9, 0, hlen, klen, 1);
+	do_test (&json_ctx, 9, 3, hlen, klen, 0);
+	do_test (&json_ctx, 9, 3, hlen, klen, 1);
+	do_test (&json_ctx, 9, 9, hlen, klen, 0);
+	do_test (&json_ctx, 9, 9, hlen, klen, 1);
+	do_test (&json_ctx, 9, 15, hlen, klen, 0);
+	do_test (&json_ctx, 9, 15, hlen, klen, 1);
+
+	do_test (&json_ctx, 15, 0, hlen, klen, 0);
+	do_test (&json_ctx, 15, 0, hlen, klen, 1);
+	do_test (&json_ctx, 15, 3, hlen, klen, 0);
+	do_test (&json_ctx, 15, 3, hlen, klen, 1);
+	do_test (&json_ctx, 15, 9, hlen, klen, 0);
+	do_test (&json_ctx, 15, 9, hlen, klen, 1);
+	do_test (&json_ctx, 15, 15, hlen, klen, 0);
+	do_test (&json_ctx, 15, 15, hlen, klen, 1);
       }
-  do_test (0, 0, page_size - 1, 16, 0);
-  do_test (0, 0, page_size - 1, 16, 1);
+  do_test (&json_ctx, 0, 0, page_size - 1, 16, 0);
+  do_test (&json_ctx, 0, 0, page_size - 1, 16, 1);
+
+  json_array_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_attr_object_end (&json_ctx);
+  json_document_end (&json_ctx);
 
   return ret;
 }
-- 
2.39.1

Re: [PATCH] build: All microbenchmarks should output in JSON. (bug 27740)

[Girish Joshi]

Ping.

Girish Joshi


On Sun, Feb 19, 2023 at 6:20 PM Girish Joshi <girish946@gmail.com> wrote:
>
> Hello,
> According to bug #27740 [1], I found that the following files need to
> be updated.
>
>     - benchtests/bench-memccpy.c
>     - benchtests/bench-strcasestr.c
>     - benchtests/bench-strcpy_chk.c
>     - benchtests/bench-strsep.c
>     - benchtests/bench-strtok.c
>
> Also, the output json needs to be fixed in two files.
>     - bench-math-inlines.c
>     - bench-strcoll.c
>
> This patch fixes both of these things.
> The output is tested using
> benchtests/scripts/compare_strings.py.
>
> Please let me know if some correction is needed in this patch.
>
> Thanks,
> Girish Joshi
> girishjoshi.io
>
> [1]: https://sourceware.org/bugzilla/show_bug.cgi?id=27740

Re: [PATCH] build: All microbenchmarks should output in JSON. (bug 27740)

[Noah Goldstein]

On Sun, Mar 5, 2023 at 12:15 AM Girish Joshi via Libc-alpha
<libc-alpha@sourceware.org> wrote:
>
> Ping.
>
> Girish Joshi
>
>
> On Sun, Feb 19, 2023 at 6:20 PM Girish Joshi <girish946@gmail.com> wrote:
> >
> > Hello,
> > According to bug #27740 [1], I found that the following files need to
> > be updated.
> >
> >     - benchtests/bench-memccpy.c
> >     - benchtests/bench-strcasestr.c
> >     - benchtests/bench-strcpy_chk.c
> >     - benchtests/bench-strsep.c
> >     - benchtests/bench-strtok.c
> >
> > Also, the output json needs to be fixed in two files.
> >     - bench-math-inlines.c
> >     - bench-strcoll.c
> >
> > This patch fixes both of these things.
> > The output is tested using
> > benchtests/scripts/compare_strings.py.
> >
> > Please let me know if some correction is needed in this patch.

When applying:
```
Applying: build: All microbenchmarks should output in JSON. (bug 27740)
.git/rebase-apply/patch:235: trailing whitespace.
    json_element_string(&json_ctx, impl->name);
warning: 1 line adds whitespace errors.
```

Also can you please post the patch as inline text in the email? It
allows patchwork
to detect it and easier to inline comments during review.

Some general feedback:
1) The changes to `benchtests/bench-math-inlines.c`/`benchtests/bench-strcoll.c`
seem unrelated, can you make that a seperate patch.
2) You are missing some test parameters in most of the files (mostly `max_char`)
also `dlen` in `bench_strcpy_chk.c`. Can you make sure you get all of them that
are relevant to the input.

> >
> > Thanks,
> > Girish Joshi
> > girishjoshi.io
> >
> > [1]: https://sourceware.org/bugzilla/show_bug.cgi?id=27740