Re: [PATCH 5/5] gdb.perf/: Add JIT performance test

[Andrew Burgess <aburgess@redhat.com>]

Ilya Leoshkevich <iii@linux.ibm.com> writes:

> The test registers and unregisters 500 functions.  The time it takes to
> process each function (the more functions there are, the slower it is)
> is written into a global array, which is then read and reported by the
> GDB script.

I'm basically happy with this, but I do have one suggestion...

> ---
>  gdb/testsuite/gdb.perf/jit-check.exp | 25 ++++++++
>  gdb/testsuite/gdb.perf/jit-check.py  | 60 ++++++++++++++++++++
>  gdb/testsuite/gdb.perf/jit-solib.c   | 28 +++++++++
>  gdb/testsuite/gdb.perf/jit.c         | 85 ++++++++++++++++++++++++++++
>  gdb/testsuite/gdb.perf/jit.exp       | 37 ++++++++++++
>  5 files changed, 235 insertions(+)
>  create mode 100644 gdb/testsuite/gdb.perf/jit-check.exp
>  create mode 100644 gdb/testsuite/gdb.perf/jit-check.py
>  create mode 100644 gdb/testsuite/gdb.perf/jit-solib.c
>  create mode 100644 gdb/testsuite/gdb.perf/jit.c
>  create mode 100644 gdb/testsuite/gdb.perf/jit.exp
>
> diff --git a/gdb/testsuite/gdb.perf/jit-check.exp b/gdb/testsuite/gdb.perf/jit-check.exp
> new file mode 100644
> index 00000000000..c124f294f7b
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/jit-check.exp
> @@ -0,0 +1,25 @@
> +# Copyright (C) 2022 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +# Benchmark registering and unregistering JITed code (run part).
> +
> +load_lib perftest.exp
> +load_lib gen-perf-test.exp
> +
> +if [skip_perf_tests] {
> +    return 0
> +}
> +
> +GenPerfTest::standard_run_driver jit.exp make_testcase_config jit-check.py JitCheck
> diff --git a/gdb/testsuite/gdb.perf/jit-check.py b/gdb/testsuite/gdb.perf/jit-check.py
> new file mode 100644
> index 00000000000..c5b46a23ffc
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/jit-check.py
> @@ -0,0 +1,60 @@
> +# Copyright (C) 2022 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +# Benchmark for registering and unregistering JITed code.
> +
> +import os
> +import shlex
> +
> +from perftest import measure
> +from perftest import perftest
> +from perftest import testresult
> +from perftest import utils
> +
> +
> +class JitCheck(perftest.TestCaseWithBasicMeasurements):
> +    def __init__(self, name, run_names, binfile):
> +        super(JitCheck, self).__init__(name)
> +        self.run_names = run_names
> +        self.binfile = binfile
> +        self.measurement = measure.MeasurementWallTime(
> +            testresult.SingleStatisticTestResult()
> +        )
> +        self.measure.measurements.append(self.measurement)
> +
> +    def warm_up(self):
> +        pass
> +
> +    def execute_test(self):
> +        for run in self.run_names:
> +            exe = "{}-{}".format(self.binfile, utils.convert_spaces(run))
> +            utils.select_file(exe)
> +
> +            # Help the binary find the shared objects.
> +            pieces = os.path.join(os.path.dirname(exe), "pieces")
> +            gdb.execute("cd {}".format(shlex.quote(pieces)))
> +
> +            # Measure the total run time.
> +            utils.runto_main()
> +            gdb.execute("tbreak done_breakpoint")
> +            self.measure.measure(lambda: gdb.execute("continue"), run)
> +
> +            # Extract and record processing times for individual functions.
> +            for array_name in ("register_times", "unregister_times"):
> +                array_value = gdb.parse_and_eval(array_name)
> +                for i in range(*array_value.type.range()):
> +                    parameter = "{}:{}:{}".format(run, array_name, i + 1)
> +                    result = int(array_value[i]) / 1e6
> +                    self.measurement.result.record(parameter, result)
> diff --git a/gdb/testsuite/gdb.perf/jit-solib.c b/gdb/testsuite/gdb.perf/jit-solib.c
> new file mode 100644
> index 00000000000..1171f1b86da
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/jit-solib.c
> @@ -0,0 +1,28 @@
> +/* This test program is part of GDB, the GNU debugger.
> +
> +   Copyright (C) 2022 Free Software Foundation, Inc.
> +
> +   This program is free software; you can redistribute it and/or modify
> +   it under the terms of the GNU General Public License as published by
> +   the Free Software Foundation; either version 3 of the License, or
> +   (at your option) any later version.
> +
> +   This program is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +   GNU General Public License for more details.
> +
> +   You should have received a copy of the GNU General Public License
> +   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
> +
> +/* Shared object for benchmarking registering and unregistering JITed code.
> +   It is simply copied into memory, so there should be no relocations.
> +   Must be compiled with -DSHLIB=<shared object number>.  */
> +
> +#define CAT_1(x, y) x##y
> +#define CAT(x, y) CAT_1 (x, y)
> +int
> +CAT (jited_func_, SHLIB) (void)
> +{
> +  return SHLIB;
> +}
> diff --git a/gdb/testsuite/gdb.perf/jit.c b/gdb/testsuite/gdb.perf/jit.c
> new file mode 100644
> index 00000000000..7fc8b4a6339
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/jit.c
> @@ -0,0 +1,85 @@
> +/* This test program is part of GDB, the GNU debugger.
> +
> +   Copyright (C) 2022 Free Software Foundation, Inc.
> +
> +   This program is free software; you can redistribute it and/or modify
> +   it under the terms of the GNU General Public License as published by
> +   the Free Software Foundation; either version 3 of the License, or
> +   (at your option) any later version.
> +
> +   This program is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +   GNU General Public License for more details.
> +
> +   You should have received a copy of the GNU General Public License
> +   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
> +
> +/* Benchmark for registering and unregistering JITed code.
> +   Must be compiled with -DSHLIB=<total number of shared objects>.
> */

Like I said on patch #3 I think we should change SHLIB to something
else, so this file would need updating.

> +
> +#include "../gdb.base/jit-elf-util.h"
> +#include "../gdb.base/jit-protocol-util.h"
> +#include <stdint.h>
> +#include <sys/time.h>
> +
> +static struct jit_code_entry entries[SHLIB];
> +static uint64_t register_times[SHLIB];
> +static uint64_t unregister_times[SHLIB];
> +
> +static uint64_t
> +time_delta (struct timeval *start_time)
> +{
> +  struct timeval end_time;
> +
> +  gettimeofday (&end_time, NULL);
> +  return (uint64_t)(end_time.tv_sec - start_time->tv_sec) * 1000000
> +	 + (end_time.tv_usec - start_time->tv_usec);
> +}
> +
> +void __attribute__ ((noinline)) done_breakpoint (void) {}
> +
> +int
> +main (void)
> +{
> +  struct timeval start_time;
> +  int i;
> +
> +  /* Load and register shared libraries.  */
> +  for (i = 0; i < SHLIB; i++)
> +    {
> +      int (*jited_func) (void);
> +      size_t obj_size;
> +      char name[32];
> +      void *addr;
> +
> +      sprintf (name, "jit-lib%d.so", i);
> +      addr = load_elf (name, &obj_size, NULL);
> +      sprintf (name, "jited_func_%d", i);
> +      jited_func = addr + (uintptr_t)load_symbol (addr, name);
> +
> +      entries[i].symfile_addr = addr;
> +      entries[i].symfile_size = obj_size;
> +      gettimeofday (&start_time, NULL);
> +      jit_push_back (&entries[i]);
> +      register_times[i] = time_delta (&start_time);
> +
> +      if (jited_func () != i)
> +	{
> +	  fprintf (stderr, "jited_func_%d () != %d\n", i, i);
> +	  return 1;
> +	}
> +    }
> +
> +  /* Now unregister them all in reverse order.  */
> +  for (i = SHLIB - 1; i >= 0; i--)
> +    {
> +      gettimeofday (&start_time, NULL);
> +      jit_pop_back ();
> +      unregister_times[i] = time_delta (&start_time);
> +    }
> +
> +  done_breakpoint ();
> +
> +  return 0;
> +}
> diff --git a/gdb/testsuite/gdb.perf/jit.exp b/gdb/testsuite/gdb.perf/jit.exp
> new file mode 100644
> index 00000000000..02923b290d9
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/jit.exp
> @@ -0,0 +1,37 @@
> +# Copyright (C) 2022 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +# Benchmark registering and unregistering JITed code (build part).
> +
> +load_lib perftest.exp
> +load_lib gen-perf-test.exp
> +
> +if [skip_perf_tests] {
> +    return 0
> +}
> +
> +proc make_testcase_config { } {
> +    set program_name "jit"
> +    array set testcase [GenPerfTest::init_testcase $program_name]
> +    set testcase(language) c
> +    set testcase(binary_extra_sources) { { jit.c } }
> +    set testcase(binary_link_with_shlibs) { 0 }
> +    set testcase(gen_shlib_extra_sources) { { jit-solib.c } }
> +    set testcase(run_names) { 500-sos }
> +    set testcase(nr_gen_shlibs) { 500 }

I think it would be nice if the '500' could be user controllable.  While
I was playing with this, I added this code before
'make_testcase_config':

  # make build-perf RUNTESTFLAGS='jit.exp SOLIB_COUNT=100'
  if ![info exists SOLIB_COUNT] {
      set SOLIB_COUNT 500
  }

And then made this change within make_testcase_config:

  -    set testcase(run_names) { 500-sos }
  -    set testcase(nr_gen_shlibs) { 500 }
  +    set name "${::SOLIB_COUNT}-sos"
  +    set testcase(run_names) [list $name]
  +    set testcase(nr_gen_shlibs)  ${::SOLIB_COUNT}

Then, like the comment says, I'd build with:

  make build-perf RUNTESTFLAGS="jit.exp SOLIB_COUNT=100"

And test with:

  make check-perf RUNTESTFLAGS="jit-check.exp SOLIB_COUNT=100" GDB_PERFTEST_MODE=run

I think we should include something like this, and make sure that the
new flag is documented, probably in the comment at the start of jit.exp.

However, before this is merged, I'd really like to understand why the
performance from gdb.base/jit-elf.exp is so different to the performance
we get from this test.  I'm assuming that it's something to do with how
the test is compiled, but so far I've not been able to figure it out.

Thanks,
Andrew



> +    return [array get testcase]
> +}
> +
> +GenPerfTest::standard_compile_driver jit.exp make_testcase_config
> -- 
> 2.35.3