* [PATCH 3/4] Mention perf test in testsuite/README
2013-10-16 7:10 [PATCH 0/4 V3] GDB Performance testing Yao Qi
2013-10-16 7:10 ` [PATCH 4/4] Test on solib load and unload Yao Qi
@ 2013-10-16 7:10 ` Yao Qi
2013-10-25 6:43 ` Doug Evans
2013-10-16 7:10 ` [PATCH 2/4] Perf test framework Yao Qi
` (3 subsequent siblings)
5 siblings, 1 reply; 24+ messages in thread
From: Yao Qi @ 2013-10-16 7:10 UTC (permalink / raw)
To: gdb-patches
gdb/testsuite:
2013-10-16 Yao Qi <yao@codesourcery.com>
* README: Mention performance tests.
---
gdb/testsuite/README | 27 +++++++++++++++++++++++++++
1 files changed, 27 insertions(+), 0 deletions(-)
diff --git a/gdb/testsuite/README b/gdb/testsuite/README
index ec91b14..bc06ae9 100644
--- a/gdb/testsuite/README
+++ b/gdb/testsuite/README
@@ -25,6 +25,29 @@ The second is to cd to the testsuite directory and invoke the DejaGnu
(The `site.exp' file contains a handful of useful variables like host
and target triplets, and pathnames.)
+Running the Performance Tests
+*****************************
+
+GDB Testsuite includes performance test cases, which are not run together
+with other test cases, because performance test cases are slow and need
+a quiet system. There are two ways to run the performance test cases.
+The first is to do `make check-perf' in the main build directory:
+
+ make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=8"
+
+The second is to cd to the testsuite directory and invoke the DejaGnu
+`runtest' command directly.
+
+ cd testsuite
+ make site.exp
+ runtest GDB_PERFORMANCE=both GDB_PERFORMANCE_TIMEOUT=4000 --directory=gdb.perf solib.exp SOLIB_COUNT=8
+
+Only "compile", "run" and "both" are valid to GDB_PERFORMANCE. They
+stand for "compile tests only", "run tests only", and "compile and run
+tests" respectively. "both" is the default. GDB_PERFORMANCE_TIMEOUT
+specify the timeout, which is 3000 in default. The result of
+performance test is appended in `testsuite/perftest.log'.
+
Testsuite Parameters
********************
@@ -315,6 +338,10 @@ Tests that exercise a specific GDB subsystem in more depth. For
instance, gdb.disasm exercises various disassemblers, while
gdb.stabs tests pathways through the stabs symbol reader.
+gdb.perf
+
+GDB performance tests.
+
Writing Tests
*************
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 4/4] Test on solib load and unload
2013-10-16 7:10 [PATCH 0/4 V3] GDB Performance testing Yao Qi
@ 2013-10-16 7:10 ` Yao Qi
2013-10-25 7:17 ` Doug Evans
2013-10-16 7:10 ` [PATCH 3/4] Mention perf test in testsuite/README Yao Qi
` (4 subsequent siblings)
5 siblings, 1 reply; 24+ messages in thread
From: Yao Qi @ 2013-10-16 7:10 UTC (permalink / raw)
To: gdb-patches
This patch is to add a test case to on the performance of GDB handling
load and unload of shared library.
In V3, there are some changes,
- Adapt to perf test framework changes.
- Measure load and unload separately.
In V2, there are some changes,
- A new proc gdb_produce_source to produce source files. I tried to
move all source file generation code out of solib.exp, but
compilation step still needs to know the generated file names. I
have to hard-code the file names in compilation step, which is not
good to me, so I give up on this moving.
- SOLIB_NUMBER -> SOLIB_COUNT
- New variable SOLIB_DLCLOSE_REVERSED_ORDER to control the order of
iterating a list of shared libs to dlclose them.
- New variable GDB_PERFORMANCE to enable these perf test cases.
- Remove dlsym call in solib.c.
- Update solib.py for the updated framework.
gdb/testsuite/
* lib/gdb.exp (gdb_produce_source): New procedure.
* gdb.perf/solib.c: New.
* gdb.perf/solib.exp: New.
* gdb.perf/solib.py: New.
---
gdb/testsuite/gdb.perf/solib.c | 78 ++++++++++++++++++++++++++++++++++++++
gdb/testsuite/gdb.perf/solib.exp | 78 ++++++++++++++++++++++++++++++++++++++
gdb/testsuite/gdb.perf/solib.py | 73 +++++++++++++++++++++++++++++++++++
gdb/testsuite/lib/gdb.exp | 16 ++++++++
4 files changed, 245 insertions(+), 0 deletions(-)
create mode 100644 gdb/testsuite/gdb.perf/solib.c
create mode 100644 gdb/testsuite/gdb.perf/solib.exp
create mode 100644 gdb/testsuite/gdb.perf/solib.py
diff --git a/gdb/testsuite/gdb.perf/solib.c b/gdb/testsuite/gdb.perf/solib.c
new file mode 100644
index 0000000..42973d6
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.c
@@ -0,0 +1,78 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+ Copyright (C) 2013 Free Software Foundation, Inc.
+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation; either version 3 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program. If not, see <http://www.gnu.org/licenses/>. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __WIN32__
+#include <windows.h>
+#define dlopen(name, mode) LoadLibrary (TEXT (name))
+# define dlsym(handle, func) GetProcAddress (handle, func)
+#define dlclose(handle) FreeLibrary (handle)
+#else
+#include <dlfcn.h>
+#endif
+
+static void **handles;
+
+void
+do_test_load (int number)
+{
+ char libname[40];
+ int i;
+
+ handles = malloc (sizeof (void *) * number);
+
+ for (i = 0; i < number; i++)
+ {
+ sprintf (libname, "solib-lib%d", i);
+ handles[i] = dlopen (libname, RTLD_LAZY);
+ if (handles[i] == NULL)
+ {
+ printf ("ERROR on dlopen %s\n", libname);
+ return;
+ }
+ }
+}
+
+void
+do_test_unload (int number)
+{
+ int i;
+
+ /* Unload shared libraries in different orders. */
+#ifndef SOLIB_DLCLOSE_REVERSED_ORDER
+ for (i = 0; i < number; i++)
+#else
+ for (i = number - 1; i >= 0; i--)
+#endif
+ dlclose (handles[i]);
+
+ free (handles);
+}
+
+static void
+end (void)
+{}
+
+int
+main (void)
+{
+ end ();
+
+ return 0;
+}
diff --git a/gdb/testsuite/gdb.perf/solib.exp b/gdb/testsuite/gdb.perf/solib.exp
new file mode 100644
index 0000000..fc66f77
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.exp
@@ -0,0 +1,78 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# This test case is to test the performance of GDB when it is handling
+# the shared libraries of inferior are loaded and unloaded.
+load_lib perftest.exp
+
+if [skip_perf_tests] {
+ return 0
+}
+
+standard_testfile .c
+set executable $testfile
+set expfile $testfile.exp
+
+# make check-perf RUNTESTFLAGS='solib.exp SOLIB_COUNT=1024'
+if ![info exists SOLIB_COUNT] {
+ set SOLIB_COUNT 128
+}
+
+PerfTest::assemble {
+ compile {
+ for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
+
+ # Produce source files.
+ set libname "solib-lib$i"
+ set src [standard_output_file $libname.c]
+ set exe [standard_output_file $libname]
+
+ gdb_produce_source $src { "int shr$i (void) {return 0;}" }
+
+ # Compile.
+ if { [gdb_compile_shlib $src $exe {debug}] != "" } {
+ untested "Couldn't compile $src."
+ return -1
+ }
+
+ # Delete object files to save some space.
+ file delete [standard_output_file "solib-lib$i.c.o"]
+ }
+
+ set compile_flags {debug shlib_load}
+ global SOLIB_DLCLOSE_REVERSED_ORDER
+
+ if [info exists SOLIB_DLCLOSE_REVERSED_ORDER] {
+ lappend compile_flags "additional_flags=-DSOLIB_DLCLOSE_REVERSED_ORDER"
+ }
+
+ if { [gdb_compile "$srcdir/$subdir/$srcfile" ${binfile} executable $compile_flags] == "" } {
+ PerfTest::compiled
+ }
+ }
+} {
+ startup_gdb {
+ clean_restart $binfile
+
+ if ![runto_main] {
+ fail "Can't run to main"
+ return -1
+ }
+ }
+} {
+ run {
+ gdb_test_no_output "python SolibLoadUnload\($SOLIB_COUNT\).run()"
+ }
+}
diff --git a/gdb/testsuite/gdb.perf/solib.py b/gdb/testsuite/gdb.perf/solib.py
new file mode 100644
index 0000000..91393b1
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.py
@@ -0,0 +1,73 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# This test case is to test the speed of GDB when it is handling the
+# shared libraries of inferior are loaded and unloaded.
+
+from perftest import perftest
+from perftest import measure
+
+class SolibLoadUnload1(perftest.TestCaseWithBasicMeasurements):
+ def __init__(self, solib_count, measure_load):
+ if measure_load:
+ name = "solib_load"
+ else:
+ name = "solib_unload"
+ # We want to measure time in this test.
+ super (SolibLoadUnload1, self).__init__ (name)
+ self.solib_count = solib_count
+ self.measure_load = measure_load
+
+ def warm_up(self):
+ do_test_load = "call do_test_load (%d)" % self.solib_count
+ do_test_unload = "call do_test_unload (%d)" % self.solib_count
+ gdb.execute(do_test_load)
+ gdb.execute(do_test_unload)
+
+ def execute_test(self):
+ num = self.solib_count
+ iteration = 5;
+
+ while num > 0 and iteration > 0:
+ # Do inferior calls to do_test_load and do_test_unload in pairs,
+ # but measure differently.
+ if self.measure_load:
+ do_test_load = "call do_test_load (%d)" % num
+ func = lambda: gdb.execute (do_test_load)
+
+ self.measure.measure(func, num)
+
+ do_test_unload = "call do_test_unload (%d)" % num
+ gdb.execute (do_test_unload)
+
+ else:
+ do_test_load = "call do_test_load (%d)" % num
+ gdb.execute (do_test_load)
+
+ do_test_unload = "call do_test_unload (%d)" % num
+ func = lambda: gdb.execute (do_test_unload)
+
+ self.measure.measure(func, num)
+
+ num = num / 2
+ iteration -= 1
+
+class SolibLoadUnload(object):
+ def __init__(self, solib_count):
+ self.solib_count = solib_count;
+
+ def run(self):
+ SolibLoadUnload1(self.solib_count, True).run()
+ SolibLoadUnload1(self.solib_count, False).run()
diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index 3efd539..28aca53 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -1796,6 +1796,22 @@ proc supports_reverse {} {
return 0
}
+# Produce source file NAME and write SOURCES into it.
+
+proc gdb_produce_source { name sources } {
+ set index 0
+ set f [open $name "w"]
+
+ while { ${index} < [llength ${sources}] } {
+ set line [lindex ${sources} ${index}]
+ set index [expr ${index} + 1]
+
+ set line [uplevel list $line]
+ puts $f $line
+ }
+ close $f
+}
+
# Return 1 if target is ILP32.
# This cannot be decided simply from looking at the target string,
# as it might depend on externally passed compiler options like -m64.
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 2/4] Perf test framework
2013-10-16 7:10 [PATCH 0/4 V3] GDB Performance testing Yao Qi
2013-10-16 7:10 ` [PATCH 4/4] Test on solib load and unload Yao Qi
2013-10-16 7:10 ` [PATCH 3/4] Mention perf test in testsuite/README Yao Qi
@ 2013-10-16 7:10 ` Yao Qi
2013-10-16 9:05 ` Yao Qi
2013-10-25 6:32 ` Doug Evans
2013-10-16 7:10 ` [PATCH 1/4] New make target 'check-perf' and new dir gdb.perf Yao Qi
` (2 subsequent siblings)
5 siblings, 2 replies; 24+ messages in thread
From: Yao Qi @ 2013-10-16 7:10 UTC (permalink / raw)
To: gdb-patches
This patch adds a basic framework to do performance testing for GDB.
perftest.py is about the test case, testresult.py is about test
results, and how are they saved. reporter.py is about how results
are reported (in what format). measure.py is about measuring the
execution of tests by a collection of measurements.
In V3, there are some changes,
- Add wall time measurement, cpu time measurement and vmsize
measurement.
- Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements,
which measures cpu time, wall time, and memory (vmsize).
- GDB_PERFORMANCE=run|compile|both to control the mode of perf
testing.
- New GDB_PERFORMANCE_TIMEOUT to specify the timeout.
- Split proc prepare to proc compile and startup.
- Disable GC while doing measurements.
In V2, there are several changes to address Doug and Sanimir's
comments.
- Add copyright header and docstring in perftest/__init__.py
- Remove config.py.
- Fix docstring format.
- Rename classes "SingleVariable" to "SingleStatistic".
- Don't extend gdb.Function in class TestCase. Add a new method run
to run the test case so that we can pass parameters to test.
- Allow to customize whether to warm up and to append test log.
- Move time measurement into test harness. Add a new class
Measurement for a specific measurement and a new class Measure to
measure them for a given test case.
- A new class ResultFactory to create instances of TestResult.
- New file lib/perftest.exp, which is to do some preparations and
cleanups to simplify each *.exp file.
- Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set.
gdb/testsuite/
* lib/perftest.exp: New.
* gdb.perf/lib/perftest/__init__.py: New.
* gdb.perf/lib/perftest/measure.py: New.
* gdb.perf/lib/perftest/perftest.py: New.
* gdb.perf/lib/perftest/reporter.py: New.
* gdb.perf/lib/perftest/testresult.py: New.
---
gdb/testsuite/gdb.perf/lib/perftest/__init__.py | 17 +++
gdb/testsuite/gdb.perf/lib/perftest/measure.py | 146 ++++++++++++++++++++
gdb/testsuite/gdb.perf/lib/perftest/perftest.py | 72 ++++++++++
gdb/testsuite/gdb.perf/lib/perftest/reporter.py | 64 +++++++++
gdb/testsuite/gdb.perf/lib/perftest/testresult.py | 57 ++++++++
gdb/testsuite/lib/perftest.exp | 148 +++++++++++++++++++++
6 files changed, 504 insertions(+), 0 deletions(-)
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
create mode 100644 gdb/testsuite/lib/perftest.exp
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/__init__.py b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
new file mode 100644
index 0000000..7739a3e
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
@@ -0,0 +1,17 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+"""
+GDB performance testing framework.
+"""
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
new file mode 100644
index 0000000..7478be5
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import gc
+
+class Measure(object):
+ """A class that measure and collect the interesting data for a given testcase.
+
+ An instance of Measure has a collection of measurements, and each
+ of them is to measure a given aspect, such as time and memory.
+ """
+
+ def __init__(self, measurements):
+ """Constructor of measure.
+
+ measurements is a collection of Measurement objects.
+ """
+
+ self.measurements = measurements
+
+ def measure(self, func, id):
+ """Measure the operations done by func with a collection of measurements."""
+ for m in self.measurements:
+ m.start(id)
+
+ # Enable GC, force GC and disable GC before running test in order to reduce
+ # the interference from GC.
+ gc.enable()
+ gc.collect()
+ gc.disable()
+
+ func()
+
+ gc.enable()
+
+ for m in self.measurements:
+ m.stop(id)
+
+ def report(self, reporter, name):
+ """Report the measured results."""
+ for m in self.measurements:
+ m.report(reporter, name)
+
+class Measurement(object):
+ """A measurement for a certain aspect."""
+
+ def __init__(self, name, result):
+ """Constructor of Measurement.
+
+ Attribute result is the TestResult associated with measurement.
+ """
+ self.name = name;
+ self.result = result
+
+ def start(self, id):
+ """Abstract method to start the measurement."""
+ raise NotImplementedError("Abstract Method:start")
+
+ def stop(self, id):
+ """Abstract method to stop the measurement.
+
+ When the measurement is stopped, we've got something, and
+ record them in result.
+ """
+ raise NotImplementedError("Abstract Method:stop.")
+
+ def report(self, reporter, name):
+ """Report the measured data by argument reporter."""
+ self.result.report(reporter, name + " " + self.name)
+
+class MeasurementCPUTime(Measurement):
+ """Measurement on CPU time."""
+ # On UNIX, time.clock() measures the amount of CPU time that has
+ # been used by the current process. On Windows it will measure
+ # wall-clock seconds elapsed since the first call to the function.
+ # Something other than time.clock() should be used to measure CPU
+ # time on Windows.
+
+ def __init__(self, result):
+ super(MeasurementCPUTime, self).__init__("cpu_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.clock()
+
+ def stop(self, id):
+ if os.name == 'nt':
+ cpu_time = 0
+ else:
+ cpu_time = time.clock() - self.start_time
+ self.result.record (id, cpu_time)
+
+class MeasurementWallTime(Measurement):
+ """Measurement on Wall time."""
+
+ def __init__(self, result):
+ super(MeasurementWallTime, self).__init__("wall_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.time()
+
+ def stop(self, id):
+ wall_time = time.time() - self.start_time
+ self.result.record (id, wall_time)
+
+class MeasurementVmSize(Measurement):
+ """Measurement on memory usage represented by VmSize."""
+
+ def __init__(self, result):
+ super(MeasurementVmSize, self).__init__("vmsize", result)
+
+ def _compute_process_memory_usage(self, key):
+ file_path = "/proc/%d/status" % os.getpid()
+ try:
+ t = open(file_path)
+ v = t.read()
+ t.close()
+ except:
+ return 0
+ i = v.index(key)
+ v = v[i:].split(None, 3)
+ if len(v) < 3:
+ return 0
+ return int(v[1])
+
+ def start(self, id):
+ foo = 0
+
+ def stop(self, id):
+ memory_used = self._compute_process_memory_usage("VmSize:")
+ self.result.record (id, memory_used)
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/perftest.py b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
new file mode 100644
index 0000000..f7b8a8d
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
@@ -0,0 +1,72 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import testresult
+import reporter
+from measure import Measure
+from measure import MeasurementCPUTime
+from measure import MeasurementWallTime
+from measure import MeasurementVmSize
+
+class TestCase(object):
+ """Base class of all performance testing cases.
+
+ Each sub-class should override methods execute_test, in which
+ several GDB operations are performed and measured by attribute
+ measure. Sub-class can also override method warm_up optionally
+ if the test case needs warm up.
+ """
+
+ def __init__(self, name, measure):
+ """Constructor of TestCase.
+
+ Construct an instance of TestCase with a name and a measure
+ which is to measure the test by several different measurements.
+ """
+
+ self.name = name
+ self.measure = measure
+
+ def execute_test(self):
+ """Abstract method to do the actual tests."""
+ raise NotImplementedError("Abstract Method.")
+
+ def warm_up(self):
+ """Do some operations to warm up the environment."""
+
+ def run(self, warm_up=True, append=True):
+ """Run this test case.
+
+ It is a template method to invoke method warm_up,
+ execute_test, and finally report the measured results.
+ If parameter warm_up is True, run method warm_up. If parameter
+ append is True, the test result will be appended instead of
+ overwritten.
+ """
+ if warm_up:
+ self.warm_up()
+
+ self.execute_test()
+ self.measure.report(reporter.TextReporter(append), self.name)
+
+class TestCaseWithBasicMeasurements(TestCase):
+ """Test case measuring CPU time, wall time and memory usage."""
+
+ def __init__(self, name):
+ result_factory = testresult.SingleStatisticResultFactory()
+ measurements = [MeasurementCPUTime(result_factory.create_result()),
+ MeasurementWallTime(result_factory.create_result()),
+ MeasurementVmSize(result_factory.create_result())]
+ super (TestCaseWithBasicMeasurements, self).__init__ (name, Measure(measurements))
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
new file mode 100644
index 0000000..a902e4b
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
@@ -0,0 +1,64 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class Reporter(object):
+ """Base class of reporter to report test results in a certain format.
+
+ Subclass, which is specific to a report format, should overwrite
+ methods report, start and end.
+ """
+
+ def __init__(self, append):
+ """Constructor of Reporter.
+
+ attribute append is used to determine whether to append or
+ overwrite log file.
+ """
+ self.append = append
+
+ def report(self, *args):
+ raise NotImplementedError("Abstract Method:report.")
+
+ def start(self):
+ """Invoked when reporting is started."""
+ raise NotImplementedError("Abstract Method:start.")
+
+ def end(self):
+ """Invoked when reporting is done.
+
+ It must be overridden to do some cleanups, such as closing file
+ descriptors.
+ """
+ raise NotImplementedError("Abstract Method:end.")
+
+class TextReporter(Reporter):
+ """Report results in a plain text file 'perftest.log'."""
+
+ def __init__(self, append):
+ super (TextReporter, self).__init__(Reporter(append))
+ self.txt_log = None
+
+ def report(self, *args):
+ self.txt_log.write(' '.join(str(arg) for arg in args))
+ self.txt_log.write('\n')
+
+ def start(self):
+ if self.append:
+ self.txt_log = open ("perftest.log", 'a+');
+ else:
+ self.txt_log = open ("perftest.log", 'w');
+
+ def end(self):
+ self.txt_log.close ()
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
new file mode 100644
index 0000000..e571f12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
@@ -0,0 +1,57 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class TestResult(object):
+ """Base class to record and report test results.
+
+ Method record is to record the results of test case, and report
+ method is to report the recorded results by a given reporter.
+ """
+
+ def record(self, parameter, result):
+ raise NotImplementedError("Abstract Method:record.")
+
+ def report(self, reporter, name):
+ """Report the test results by reporter."""
+ raise NotImplementedError("Abstract Method:report.")
+
+class SingleStatisticTestResult(TestResult):
+ """Test results for the test case with a single statistic."""
+
+ def __init__(self):
+ super (SingleStatisticTestResult, self).__init__ ()
+ self.results = dict ()
+
+ def record(self, parameter, result):
+ self.results[parameter] = result
+
+ def report(self, reporter, name):
+ reporter.start()
+ for key in sorted(self.results.iterkeys()):
+ reporter.report(name, key, self.results[key])
+ reporter.end()
+
+class ResultFactory(object):
+ """A factory to create an instance of TestResult."""
+
+ def create_result(self):
+ """Create an instance of TestResult."""
+ raise NotImplementedError("Abstract Method:create_result.")
+
+class SingleStatisticResultFactory(ResultFactory):
+ """A factory to create an instance of SingleStatisticTestResult."""
+
+ def create_result(self):
+ return SingleStatisticTestResult()
diff --git a/gdb/testsuite/lib/perftest.exp b/gdb/testsuite/lib/perftest.exp
new file mode 100644
index 0000000..59b204d
--- /dev/null
+++ b/gdb/testsuite/lib/perftest.exp
@@ -0,0 +1,148 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+namespace eval PerfTest {
+ # The name of python file on build.
+ variable remote_python_file
+
+ # The source files are compiled successfully or not.
+ variable compiled_ok
+
+ # A private method to set up GDB for performance testing.
+ proc _setup_perftest {} {
+ variable remote_python_file
+ global srcdir subdir testfile
+
+ set remote_python_file [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py]
+
+ # Set sys.path for module perftest.
+ gdb_test_no_output "python import os, sys"
+ gdb_test_no_output "python sys.path.insert\(0, os.path.abspath\(\"${srcdir}/${subdir}/lib\"\)\)"
+ gdb_test_no_output "python exec (open ('${remote_python_file}').read ())"
+ }
+
+ # A private method to do some cleanups when performance test is
+ # finished.
+ proc _teardown_perftest {} {
+ variable remote_python_file
+
+ remote_file host delete $remote_python_file
+ }
+
+ # Compile source files of test case. BODY is the tcl code to do
+ # actual compilation and it should invoke 'PerfTest::compiled' if
+ # compilation is successful.
+ proc compile {body} {
+ global GDB_PERFORMANCE
+
+ if { [info exists GDB_PERFORMANCE]
+ && [string compare $GDB_PERFORMANCE "run"] } {
+ variable compiled_ok
+
+ set compiled_ok 0
+ uplevel 2 $body
+
+ if {!$compiled_ok} {
+ untested "Could not compile source files."
+ return -1
+ }
+ }
+ }
+
+ # Mark the compilation is finished successfully.
+ proc compiled {} {
+ variable compiled_ok
+
+ set compiled_ok 1
+ }
+
+ # Start up GDB.
+ proc startup_gdb {body} {
+ variable compiled_ok
+
+ if {!$compiled_ok} {
+ return
+ }
+
+ uplevel 2 $body
+ }
+
+ # Run the performance test.
+ proc run {body} {
+ global timeout
+ global GDB_PERFORMANCE_TIMEOUT
+
+ set oldtimeout $timeout
+ if { [info exists GDB_PERFORMANCE_TIMEOUT] } {
+ set timeout $GDB_PERFORMANCE_TIMEOUT
+ } else {
+ set timeout 3000
+ }
+ uplevel 2 $body
+
+ set timeout $oldtimeout
+ }
+
+ # The top-level interface to PerfTest.
+ # COMPILE is the tcl code to generate and compile source files.
+ # STARTUP is the tcl code to start up GDB.
+ # RUN is the tcl code to drive GDB to do some operations.
+ proc assemble {compile startup run} {
+ variable compiled_ok
+ global GDB_PERFORMANCE
+
+ set compiled_ok 1
+ eval $compile
+
+ if {!$compiled_ok} {
+ return
+ }
+
+ # Don't execute the run if GDB_PERFORMANCE=compile.
+ if { [info exists GDB_PERFORMANCE]
+ && [string compare $GDB_PERFORMANCE "compile"] == 0} {
+ return
+ }
+
+ eval $startup
+
+ _setup_perftest
+
+ eval $run
+
+ _teardown_perftest
+ }
+}
+
+# Return true if performance tests are skipped.
+
+proc skip_perf_tests { } {
+ global GDB_PERFORMANCE
+
+ if [info exists GDB_PERFORMANCE] {
+
+ if { [string compare $GDB_PERFORMANCE "compile"]
+ && [string compare $GDB_PERFORMANCE "run"]
+ && [string compare $GDB_PERFORMANCE "both"] } {
+ # GDB_PERFORMANCE=compile|run|both is allowed.
+ unsupported "Unknown value of GDB_PERFORMANCE."
+ return 1
+ }
+
+ return 0
+ }
+
+ return 1
+}
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 0/4 V3] GDB Performance testing
@ 2013-10-16 7:10 Yao Qi
2013-10-16 7:10 ` [PATCH 4/4] Test on solib load and unload Yao Qi
` (5 more replies)
0 siblings, 6 replies; 24+ messages in thread
From: Yao Qi @ 2013-10-16 7:10 UTC (permalink / raw)
To: gdb-patches
Hi,
Here is the V3 of GDB performance testing. The changes in V3 can be
found in each patch, and they address all review comments.
The basic usages of performance testing are unchanged,
$ make check-perf
$ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver solib.exp"
$ make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=512"
We can skip compilation step like,
$ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFORMANCE=run'
or only compile test cases,
$ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFORMANCE=compile'
in default, GDB_PERFORMANCE=both.
If the perf test case takes much time, we can specify timeout by
GDB_PERFORMANCE_TIMEOUT=1000.
$ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver GDB_PERFORMANCE_TIMEOUT=4000 solib.exp"
Looks the perf test framework is in shape, so I add the patch to
testsuite/README to describe perf test.
The output in testsuite/perftest.log is like this:
solib_load cpu_time 32 0.05
solib_load cpu_time 64 0.13
solib_load cpu_time 128 0.36
solib_load cpu_time 256 1.16
solib_load cpu_time 512 4.93
solib_load wall_time 32 0.0982959270477
solib_load wall_time 64 0.261036157608
solib_load wall_time 128 0.848724126816
solib_load wall_time 256 3.21620082855
solib_load wall_time 512 15.2153270245
solib_load vmsize 32 94632
solib_load vmsize 64 94652
solib_load vmsize 128 94696
solib_load vmsize 256 94780
solib_load vmsize 512 110664
solib_unload cpu_time 32 0.03
solib_unload cpu_time 64 0.08
solib_unload cpu_time 128 0.22
solib_unload cpu_time 256 0.74
solib_unload cpu_time 512 2.64
solib_unload wall_time 32 0.0625808238983
solib_unload wall_time 64 0.183571815491
solib_unload wall_time 128 0.655772924423
solib_unload wall_time 256 2.67978000641
solib_unload wall_time 512 12.8180861473
solib_unload vmsize 32 109648
solib_unload vmsize 64 109648
solib_unload vmsize 128 109648
solib_unload vmsize 256 109648
solib_unload vmsize 512 110440
*** BLURB HERE ***
Yao Qi (4):
New make target 'check-perf' and new dir gdb.perf
Perf test framework
Mention perf test in testsuite/README
Test on solib load and unload
gdb/Makefile.in | 8 +
gdb/testsuite/Makefile.in | 4 +
gdb/testsuite/README | 27 ++++
gdb/testsuite/configure | 3 +-
gdb/testsuite/configure.ac | 2 +-
gdb/testsuite/gdb.perf/Makefile.in | 15 ++
gdb/testsuite/gdb.perf/lib/perftest/__init__.py | 17 +++
gdb/testsuite/gdb.perf/lib/perftest/measure.py | 146 ++++++++++++++++++++
gdb/testsuite/gdb.perf/lib/perftest/perftest.py | 72 ++++++++++
gdb/testsuite/gdb.perf/lib/perftest/reporter.py | 64 +++++++++
gdb/testsuite/gdb.perf/lib/perftest/testresult.py | 57 ++++++++
gdb/testsuite/gdb.perf/solib.c | 78 +++++++++++
gdb/testsuite/gdb.perf/solib.exp | 78 +++++++++++
gdb/testsuite/gdb.perf/solib.py | 73 ++++++++++
gdb/testsuite/lib/gdb.exp | 16 +++
gdb/testsuite/lib/perftest.exp | 148 +++++++++++++++++++++
16 files changed, 806 insertions(+), 2 deletions(-)
create mode 100644 gdb/testsuite/gdb.perf/Makefile.in
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
create mode 100644 gdb/testsuite/gdb.perf/solib.c
create mode 100644 gdb/testsuite/gdb.perf/solib.exp
create mode 100644 gdb/testsuite/gdb.perf/solib.py
create mode 100644 gdb/testsuite/lib/perftest.exp
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 1/4] New make target 'check-perf' and new dir gdb.perf
2013-10-16 7:10 [PATCH 0/4 V3] GDB Performance testing Yao Qi
` (2 preceding siblings ...)
2013-10-16 7:10 ` [PATCH 2/4] Perf test framework Yao Qi
@ 2013-10-16 7:10 ` Yao Qi
2013-10-25 3:04 ` Doug Evans
2013-10-24 2:44 ` ping: [PATCH 0/4 V3] GDB Performance testing Yao Qi
2013-10-25 2:59 ` Doug Evans
5 siblings, 1 reply; 24+ messages in thread
From: Yao Qi @ 2013-10-16 7:10 UTC (permalink / raw)
To: gdb-patches
We add a new dir gdb.perf in testsuite for all performance tests.
However, current 'make check' logic will either run dejagnu in
directory testsuite or iterate all gdb.* directories which has *.exp
files. Both of them will run tests in gdb.perf. We want to achieve:
1) typical 'make check' should not run performance tests. In each perf
test case, GDB_PERFORMANCE is checked. If it doesn't exist, return.
2) run perf tests easily. We add a new makefile target 'check-perf'.
V2 is simpler than V1, since we don't have to filter out gdb.perf
directory.
V3: Move GDB_PERFORMANCE=both appears before RUNTESTFLAGS. It was
approved by Doug.
gdb:
2013-10-16 Yao Qi <yao@codesourcery.com>
* Makefile.in (check-perf): New target.
gdb/testsuite:
2013-10-16 Yao Qi <yao@codesourcery.com>
* Makefile.in (check-perf): New target.
* configure.ac (AC_OUTPUT): Output Makefile in gdb.perf.
* configure: Re-generated.
* gdb.perf/Makefile.in: New.
---
gdb/Makefile.in | 8 ++++++++
gdb/testsuite/Makefile.in | 4 ++++
gdb/testsuite/configure | 3 ++-
gdb/testsuite/configure.ac | 2 +-
gdb/testsuite/gdb.perf/Makefile.in | 15 +++++++++++++++
5 files changed, 30 insertions(+), 2 deletions(-)
create mode 100644 gdb/testsuite/gdb.perf/Makefile.in
diff --git a/gdb/Makefile.in b/gdb/Makefile.in
index f13e5c6..05ae637 100644
--- a/gdb/Makefile.in
+++ b/gdb/Makefile.in
@@ -1009,6 +1009,14 @@ check: force
$(MAKE) $(TARGET_FLAGS_TO_PASS) check; \
else true; fi
+check-perf: force
+ @if [ -f testsuite/Makefile ]; then \
+ rootme=`pwd`; export rootme; \
+ rootsrc=`cd $(srcdir); pwd`; export rootsrc; \
+ cd testsuite; \
+ $(MAKE) $(TARGET_FLAGS_TO_PASS) check-perf; \
+ else true; fi
+
# The idea is to parallelize testing of multilibs, for example:
# make -j3 check//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu}
# will run 3 concurrent sessions of check, eventually testing all 10
diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in
index a7b3d5c..db32b4c 100644
--- a/gdb/testsuite/Makefile.in
+++ b/gdb/testsuite/Makefile.in
@@ -187,6 +187,10 @@ check-gdb.base%: all $(abs_builddir)/site.exp
@if test ! -d gdb.base$*; then mkdir gdb.base$*; fi
$(DO_RUNTEST) $(BASE$*_FILES) --outdir gdb.base$* $(RUNTESTFLAGS)
+check-perf: all $(abs_builddir)/site.exp
+ @if test ! -d gdb.perf; then mkdir gdb.perf; fi
+ $(DO_RUNTEST) --directory=gdb.perf --outdir gdb.perf GDB_PERFORMANCE=both $(RUNTESTFLAGS)
+
subdir_do: force
@for i in $(DODIRS); do \
if [ -d ./$$i ] ; then \
diff --git a/gdb/testsuite/configure b/gdb/testsuite/configure
index a40c144..da590f3 100755
--- a/gdb/testsuite/configure
+++ b/gdb/testsuite/configure
@@ -3448,7 +3448,7 @@ done
-ac_config_files="$ac_config_files Makefile gdb.ada/Makefile gdb.arch/Makefile gdb.asm/Makefile gdb.base/Makefile gdb.btrace/Makefile gdb.cell/Makefile gdb.cp/Makefile gdb.disasm/Makefile gdb.dwarf2/Makefile gdb.fortran/Makefile gdb.go/Makefile gdb.server/Makefile gdb.java/Makefile gdb.hp/Makefile gdb.hp/gdb.objdbg/Makefile gdb.hp/gdb.base-hp/Makefile gdb.hp/gdb.aCC/Makefile gdb.hp/gdb.compat/Makefile gdb.hp/gdb.defects/Makefile gdb.linespec/Makefile gdb.mi/Makefile gdb.modula2/Makefile gdb.multi/Makefile gdb.objc/Makefile gdb.opencl/Makefile gdb.opt/Makefile gdb.pascal/Makefile gdb.python/Makefile gdb.reverse/Makefile gdb.stabs/Makefile gdb.threads/Makefile gdb.trace/Makefile gdb.xml/Makefile"
+ac_config_files="$ac_config_files Makefile gdb.ada/Makefile gdb.arch/Makefile gdb.asm/Makefile gdb.base/Makefile gdb.btrace/Makefile gdb.cell/Makefile gdb.cp/Makefile gdb.disasm/Makefile gdb.dwarf2/Makefile gdb.fortran/Makefile gdb.go/Makefile gdb.server/Makefile gdb.java/Makefile gdb.hp/Makefile gdb.hp/gdb.objdbg/Makefile gdb.hp/gdb.base-hp/Makefile gdb.hp/gdb.aCC/Makefile gdb.hp/gdb.compat/Makefile gdb.hp/gdb.defects/Makefile gdb.linespec/Makefile gdb.mi/Makefile gdb.modula2/Makefile gdb.multi/Makefile gdb.objc/Makefile gdb.opencl/Makefile gdb.opt/Makefile gdb.pascal/Makefile gdb.perf/Makefile gdb.python/Makefile gdb.reverse/Makefile gdb.stabs/Makefile gdb.threads/Makefile gdb.trace/Makefile gdb.xml/Makefile"
cat >confcache <<\_ACEOF
# This file is a shell script that caches the results of configure
@@ -4176,6 +4176,7 @@ do
"gdb.opencl/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.opencl/Makefile" ;;
"gdb.opt/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.opt/Makefile" ;;
"gdb.pascal/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.pascal/Makefile" ;;
+ "gdb.perf/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.perf/Makefile" ;;
"gdb.python/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.python/Makefile" ;;
"gdb.reverse/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.reverse/Makefile" ;;
"gdb.stabs/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.stabs/Makefile" ;;
diff --git a/gdb/testsuite/configure.ac b/gdb/testsuite/configure.ac
index 9e07021..94f96cc 100644
--- a/gdb/testsuite/configure.ac
+++ b/gdb/testsuite/configure.ac
@@ -97,5 +97,5 @@ AC_OUTPUT([Makefile \
gdb.hp/gdb.defects/Makefile gdb.linespec/Makefile \
gdb.mi/Makefile gdb.modula2/Makefile gdb.multi/Makefile \
gdb.objc/Makefile gdb.opencl/Makefile gdb.opt/Makefile gdb.pascal/Makefile \
- gdb.python/Makefile gdb.reverse/Makefile gdb.stabs/Makefile \
+ gdb.perf/Makefile gdb.python/Makefile gdb.reverse/Makefile gdb.stabs/Makefile \
gdb.threads/Makefile gdb.trace/Makefile gdb.xml/Makefile])
diff --git a/gdb/testsuite/gdb.perf/Makefile.in b/gdb/testsuite/gdb.perf/Makefile.in
new file mode 100644
index 0000000..2071d12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/Makefile.in
@@ -0,0 +1,15 @@
+VPATH = @srcdir@
+srcdir = @srcdir@
+
+.PHONY: all clean mostlyclean distclean realclean
+
+PROGS =
+
+all info install-info dvi install uninstall installcheck check:
+ @echo "Nothing to be done for $@..."
+
+clean mostlyclean:
+ -rm -f *.o *.diff *~ core $(PROGS)
+
+distclean maintainer-clean realclean: clean
+ -rm -f Makefile config.status config.log gdb.log gdb.sum
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-10-16 7:10 ` [PATCH 2/4] Perf test framework Yao Qi
@ 2013-10-16 9:05 ` Yao Qi
2013-10-24 14:20 ` Yao Qi
2013-10-25 6:33 ` Doug Evans
2013-10-25 6:32 ` Doug Evans
1 sibling, 2 replies; 24+ messages in thread
From: Yao Qi @ 2013-10-16 9:05 UTC (permalink / raw)
To: gdb-patches
On 10/16/2013 03:09 PM, Yao Qi wrote:
> + def start(self, id):
> + foo = 0
Missed the change during git rebase. "pass" should be used here.
Here is an updated one.
--
Yao (é½å°§)
gdb/testsuite/
* lib/perftest.exp: New.
* gdb.perf/lib/perftest/__init__.py: New.
* gdb.perf/lib/perftest/measure.py: New.
* gdb.perf/lib/perftest/perftest.py: New.
* gdb.perf/lib/perftest/reporter.py: New.
* gdb.perf/lib/perftest/testresult.py: New.
---
gdb/testsuite/gdb.perf/lib/perftest/__init__.py | 17 +++
gdb/testsuite/gdb.perf/lib/perftest/measure.py | 146 ++++++++++++++++++++
gdb/testsuite/gdb.perf/lib/perftest/perftest.py | 72 ++++++++++
gdb/testsuite/gdb.perf/lib/perftest/reporter.py | 64 +++++++++
gdb/testsuite/gdb.perf/lib/perftest/testresult.py | 57 ++++++++
gdb/testsuite/lib/perftest.exp | 148 +++++++++++++++++++++
6 files changed, 504 insertions(+), 0 deletions(-)
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
create mode 100644 gdb/testsuite/lib/perftest.exp
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/__init__.py b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
new file mode 100644
index 0000000..7739a3e
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
@@ -0,0 +1,17 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+"""
+GDB performance testing framework.
+"""
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
new file mode 100644
index 0000000..8c3dc47
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import gc
+
+class Measure(object):
+ """A class that measure and collect the interesting data for a given testcase.
+
+ An instance of Measure has a collection of measurements, and each
+ of them is to measure a given aspect, such as time and memory.
+ """
+
+ def __init__(self, measurements):
+ """Constructor of measure.
+
+ measurements is a collection of Measurement objects.
+ """
+
+ self.measurements = measurements
+
+ def measure(self, func, id):
+ """Measure the operations done by func with a collection of measurements."""
+ for m in self.measurements:
+ m.start(id)
+
+ # Enable GC, force GC and disable GC before running test in order to reduce
+ # the interference from GC.
+ gc.enable()
+ gc.collect()
+ gc.disable()
+
+ func()
+
+ gc.enable()
+
+ for m in self.measurements:
+ m.stop(id)
+
+ def report(self, reporter, name):
+ """Report the measured results."""
+ for m in self.measurements:
+ m.report(reporter, name)
+
+class Measurement(object):
+ """A measurement for a certain aspect."""
+
+ def __init__(self, name, result):
+ """Constructor of Measurement.
+
+ Attribute result is the TestResult associated with measurement.
+ """
+ self.name = name;
+ self.result = result
+
+ def start(self, id):
+ """Abstract method to start the measurement."""
+ raise NotImplementedError("Abstract Method:start")
+
+ def stop(self, id):
+ """Abstract method to stop the measurement.
+
+ When the measurement is stopped, we've got something, and
+ record them in result.
+ """
+ raise NotImplementedError("Abstract Method:stop.")
+
+ def report(self, reporter, name):
+ """Report the measured data by argument reporter."""
+ self.result.report(reporter, name + " " + self.name)
+
+class MeasurementCPUTime(Measurement):
+ """Measurement on CPU time."""
+ # On UNIX, time.clock() measures the amount of CPU time that has
+ # been used by the current process. On Windows it will measure
+ # wall-clock seconds elapsed since the first call to the function.
+ # Something other than time.clock() should be used to measure CPU
+ # time on Windows.
+
+ def __init__(self, result):
+ super(MeasurementCPUTime, self).__init__("cpu_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.clock()
+
+ def stop(self, id):
+ if os.name == 'nt':
+ cpu_time = 0
+ else:
+ cpu_time = time.clock() - self.start_time
+ self.result.record (id, cpu_time)
+
+class MeasurementWallTime(Measurement):
+ """Measurement on Wall time."""
+
+ def __init__(self, result):
+ super(MeasurementWallTime, self).__init__("wall_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.time()
+
+ def stop(self, id):
+ wall_time = time.time() - self.start_time
+ self.result.record (id, wall_time)
+
+class MeasurementVmSize(Measurement):
+ """Measurement on memory usage represented by VmSize."""
+
+ def __init__(self, result):
+ super(MeasurementVmSize, self).__init__("vmsize", result)
+
+ def _compute_process_memory_usage(self, key):
+ file_path = "/proc/%d/status" % os.getpid()
+ try:
+ t = open(file_path)
+ v = t.read()
+ t.close()
+ except:
+ return 0
+ i = v.index(key)
+ v = v[i:].split(None, 3)
+ if len(v) < 3:
+ return 0
+ return int(v[1])
+
+ def start(self, id):
+ pass
+
+ def stop(self, id):
+ memory_used = self._compute_process_memory_usage("VmSize:")
+ self.result.record (id, memory_used)
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/perftest.py b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
new file mode 100644
index 0000000..f7b8a8d
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
@@ -0,0 +1,72 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import testresult
+import reporter
+from measure import Measure
+from measure import MeasurementCPUTime
+from measure import MeasurementWallTime
+from measure import MeasurementVmSize
+
+class TestCase(object):
+ """Base class of all performance testing cases.
+
+ Each sub-class should override methods execute_test, in which
+ several GDB operations are performed and measured by attribute
+ measure. Sub-class can also override method warm_up optionally
+ if the test case needs warm up.
+ """
+
+ def __init__(self, name, measure):
+ """Constructor of TestCase.
+
+ Construct an instance of TestCase with a name and a measure
+ which is to measure the test by several different measurements.
+ """
+
+ self.name = name
+ self.measure = measure
+
+ def execute_test(self):
+ """Abstract method to do the actual tests."""
+ raise NotImplementedError("Abstract Method.")
+
+ def warm_up(self):
+ """Do some operations to warm up the environment."""
+
+ def run(self, warm_up=True, append=True):
+ """Run this test case.
+
+ It is a template method to invoke method warm_up,
+ execute_test, and finally report the measured results.
+ If parameter warm_up is True, run method warm_up. If parameter
+ append is True, the test result will be appended instead of
+ overwritten.
+ """
+ if warm_up:
+ self.warm_up()
+
+ self.execute_test()
+ self.measure.report(reporter.TextReporter(append), self.name)
+
+class TestCaseWithBasicMeasurements(TestCase):
+ """Test case measuring CPU time, wall time and memory usage."""
+
+ def __init__(self, name):
+ result_factory = testresult.SingleStatisticResultFactory()
+ measurements = [MeasurementCPUTime(result_factory.create_result()),
+ MeasurementWallTime(result_factory.create_result()),
+ MeasurementVmSize(result_factory.create_result())]
+ super (TestCaseWithBasicMeasurements, self).__init__ (name, Measure(measurements))
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
new file mode 100644
index 0000000..a902e4b
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
@@ -0,0 +1,64 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class Reporter(object):
+ """Base class of reporter to report test results in a certain format.
+
+ Subclass, which is specific to a report format, should overwrite
+ methods report, start and end.
+ """
+
+ def __init__(self, append):
+ """Constructor of Reporter.
+
+ attribute append is used to determine whether to append or
+ overwrite log file.
+ """
+ self.append = append
+
+ def report(self, *args):
+ raise NotImplementedError("Abstract Method:report.")
+
+ def start(self):
+ """Invoked when reporting is started."""
+ raise NotImplementedError("Abstract Method:start.")
+
+ def end(self):
+ """Invoked when reporting is done.
+
+ It must be overridden to do some cleanups, such as closing file
+ descriptors.
+ """
+ raise NotImplementedError("Abstract Method:end.")
+
+class TextReporter(Reporter):
+ """Report results in a plain text file 'perftest.log'."""
+
+ def __init__(self, append):
+ super (TextReporter, self).__init__(Reporter(append))
+ self.txt_log = None
+
+ def report(self, *args):
+ self.txt_log.write(' '.join(str(arg) for arg in args))
+ self.txt_log.write('\n')
+
+ def start(self):
+ if self.append:
+ self.txt_log = open ("perftest.log", 'a+');
+ else:
+ self.txt_log = open ("perftest.log", 'w');
+
+ def end(self):
+ self.txt_log.close ()
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
new file mode 100644
index 0000000..e571f12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
@@ -0,0 +1,57 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class TestResult(object):
+ """Base class to record and report test results.
+
+ Method record is to record the results of test case, and report
+ method is to report the recorded results by a given reporter.
+ """
+
+ def record(self, parameter, result):
+ raise NotImplementedError("Abstract Method:record.")
+
+ def report(self, reporter, name):
+ """Report the test results by reporter."""
+ raise NotImplementedError("Abstract Method:report.")
+
+class SingleStatisticTestResult(TestResult):
+ """Test results for the test case with a single statistic."""
+
+ def __init__(self):
+ super (SingleStatisticTestResult, self).__init__ ()
+ self.results = dict ()
+
+ def record(self, parameter, result):
+ self.results[parameter] = result
+
+ def report(self, reporter, name):
+ reporter.start()
+ for key in sorted(self.results.iterkeys()):
+ reporter.report(name, key, self.results[key])
+ reporter.end()
+
+class ResultFactory(object):
+ """A factory to create an instance of TestResult."""
+
+ def create_result(self):
+ """Create an instance of TestResult."""
+ raise NotImplementedError("Abstract Method:create_result.")
+
+class SingleStatisticResultFactory(ResultFactory):
+ """A factory to create an instance of SingleStatisticTestResult."""
+
+ def create_result(self):
+ return SingleStatisticTestResult()
diff --git a/gdb/testsuite/lib/perftest.exp b/gdb/testsuite/lib/perftest.exp
new file mode 100644
index 0000000..59b204d
--- /dev/null
+++ b/gdb/testsuite/lib/perftest.exp
@@ -0,0 +1,148 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+namespace eval PerfTest {
+ # The name of python file on build.
+ variable remote_python_file
+
+ # The source files are compiled successfully or not.
+ variable compiled_ok
+
+ # A private method to set up GDB for performance testing.
+ proc _setup_perftest {} {
+ variable remote_python_file
+ global srcdir subdir testfile
+
+ set remote_python_file [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py]
+
+ # Set sys.path for module perftest.
+ gdb_test_no_output "python import os, sys"
+ gdb_test_no_output "python sys.path.insert\(0, os.path.abspath\(\"${srcdir}/${subdir}/lib\"\)\)"
+ gdb_test_no_output "python exec (open ('${remote_python_file}').read ())"
+ }
+
+ # A private method to do some cleanups when performance test is
+ # finished.
+ proc _teardown_perftest {} {
+ variable remote_python_file
+
+ remote_file host delete $remote_python_file
+ }
+
+ # Compile source files of test case. BODY is the tcl code to do
+ # actual compilation and it should invoke 'PerfTest::compiled' if
+ # compilation is successful.
+ proc compile {body} {
+ global GDB_PERFORMANCE
+
+ if { [info exists GDB_PERFORMANCE]
+ && [string compare $GDB_PERFORMANCE "run"] } {
+ variable compiled_ok
+
+ set compiled_ok 0
+ uplevel 2 $body
+
+ if {!$compiled_ok} {
+ untested "Could not compile source files."
+ return -1
+ }
+ }
+ }
+
+ # Mark the compilation is finished successfully.
+ proc compiled {} {
+ variable compiled_ok
+
+ set compiled_ok 1
+ }
+
+ # Start up GDB.
+ proc startup_gdb {body} {
+ variable compiled_ok
+
+ if {!$compiled_ok} {
+ return
+ }
+
+ uplevel 2 $body
+ }
+
+ # Run the performance test.
+ proc run {body} {
+ global timeout
+ global GDB_PERFORMANCE_TIMEOUT
+
+ set oldtimeout $timeout
+ if { [info exists GDB_PERFORMANCE_TIMEOUT] } {
+ set timeout $GDB_PERFORMANCE_TIMEOUT
+ } else {
+ set timeout 3000
+ }
+ uplevel 2 $body
+
+ set timeout $oldtimeout
+ }
+
+ # The top-level interface to PerfTest.
+ # COMPILE is the tcl code to generate and compile source files.
+ # STARTUP is the tcl code to start up GDB.
+ # RUN is the tcl code to drive GDB to do some operations.
+ proc assemble {compile startup run} {
+ variable compiled_ok
+ global GDB_PERFORMANCE
+
+ set compiled_ok 1
+ eval $compile
+
+ if {!$compiled_ok} {
+ return
+ }
+
+ # Don't execute the run if GDB_PERFORMANCE=compile.
+ if { [info exists GDB_PERFORMANCE]
+ && [string compare $GDB_PERFORMANCE "compile"] == 0} {
+ return
+ }
+
+ eval $startup
+
+ _setup_perftest
+
+ eval $run
+
+ _teardown_perftest
+ }
+}
+
+# Return true if performance tests are skipped.
+
+proc skip_perf_tests { } {
+ global GDB_PERFORMANCE
+
+ if [info exists GDB_PERFORMANCE] {
+
+ if { [string compare $GDB_PERFORMANCE "compile"]
+ && [string compare $GDB_PERFORMANCE "run"]
+ && [string compare $GDB_PERFORMANCE "both"] } {
+ # GDB_PERFORMANCE=compile|run|both is allowed.
+ unsupported "Unknown value of GDB_PERFORMANCE."
+ return 1
+ }
+
+ return 0
+ }
+
+ return 1
+}
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
* ping: [PATCH 0/4 V3] GDB Performance testing
2013-10-16 7:10 [PATCH 0/4 V3] GDB Performance testing Yao Qi
` (3 preceding siblings ...)
2013-10-16 7:10 ` [PATCH 1/4] New make target 'check-perf' and new dir gdb.perf Yao Qi
@ 2013-10-24 2:44 ` Yao Qi
2013-10-24 8:50 ` Agovic, Sanimir
2013-10-25 2:59 ` Doug Evans
5 siblings, 1 reply; 24+ messages in thread
From: Yao Qi @ 2013-10-24 2:44 UTC (permalink / raw)
To: gdb-patches
On 10/16/2013 03:09 PM, Yao Qi wrote:
> Here is the V3 of GDB performance testing. The changes in V3 can be
> found in each patch, and they address all review comments.
>
> The basic usages of performance testing are unchanged,
>
> $ make check-perf
> $ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver solib.exp"
> $ make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=512"
>
> We can skip compilation step like,
>
> $ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFORMANCE=run'
>
> or only compile test cases,
>
> $ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFORMANCE=compile'
>
> in default, GDB_PERFORMANCE=both.
>
> If the perf test case takes much time, we can specify timeout by
> GDB_PERFORMANCE_TIMEOUT=1000.
>
> $ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver GDB_PERFORMANCE_TIMEOUT=4000 solib.exp"
>
> Looks the perf test framework is in shape, so I add the patch to
> testsuite/README to describe perf test.
Ping. https://sourceware.org/ml/gdb-patches/2013-10/msg00470.html
--
Yao (é½å°§)
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: ping: [PATCH 0/4 V3] GDB Performance testing
2013-10-24 2:44 ` ping: [PATCH 0/4 V3] GDB Performance testing Yao Qi
@ 2013-10-24 8:50 ` Agovic, Sanimir
2013-10-24 14:13 ` Yao Qi
0 siblings, 1 reply; 24+ messages in thread
From: Agovic, Sanimir @ 2013-10-24 8:50 UTC (permalink / raw)
To: 'Yao Qi'; +Cc: gdb-patches
Looks good to me, looking forward to your commit.
A minor comment in your perf framework:
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
new file mode 100644
index 0000000..7478be5
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import gc
+
+class Measure(object):
+ """A class that measure and collect the interesting data for a given testcase.
+
+ An instance of Measure has a collection of measurements, and each
+ of them is to measure a given aspect, such as time and memory.
+ """
+
+ def __init__(self, measurements):
+ """Constructor of measure.
+
+ measurements is a collection of Measurement objects.
+ """
+
+ self.measurements = measurements
+
+ def measure(self, func, id):
+ """Measure the operations done by func with a collection of measurements."""
+ for m in self.measurements:
+ m.start(id)
+
+ # Enable GC, force GC and disable GC before running test in order to reduce
+ # the interference from GC.
+ gc.enable()
+ gc.collect()
+ gc.disable()
+
+ func()
+
+ gc.enable()
I`d rather exclude gc cycles completely from the measurements or include gc collection
_after_ you run your perf test. Otherwise the results will depend on the order the perf
tests are run.
-Sanimir
> -----Original Message-----
> From: gdb-patches-owner@sourceware.org [mailto:gdb-patches-owner@sourceware.org] On Behalf
> Of Yao Qi
> Sent: Thursday, October 24, 2013 04:43 AM
> To: gdb-patches@sourceware.org
> Subject: ping: [PATCH 0/4 V3] GDB Performance testing
>
> On 10/16/2013 03:09 PM, Yao Qi wrote:
> > Here is the V3 of GDB performance testing. The changes in V3 can be
> > found in each patch, and they address all review comments.
> >
> > The basic usages of performance testing are unchanged,
> >
> > $ make check-perf
> > $ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver solib.exp"
> > $ make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=512"
> >
> > We can skip compilation step like,
> >
> > $ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFORMANCE=run'
> >
> > or only compile test cases,
> >
> > $ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFORMANCE=compile'
> >
> > in default, GDB_PERFORMANCE=both.
> >
> > If the perf test case takes much time, we can specify timeout by
> > GDB_PERFORMANCE_TIMEOUT=1000.
> >
> > $ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver
> GDB_PERFORMANCE_TIMEOUT=4000 solib.exp"
> >
> > Looks the perf test framework is in shape, so I add the patch to
> > testsuite/README to describe perf test.
>
> Ping. https://sourceware.org/ml/gdb-patches/2013-10/msg00470.html
>
> --
> Yao (齐尧)
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen, Deutschland
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
Registergericht: Muenchen HRB 47456
Ust.-IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: ping: [PATCH 0/4 V3] GDB Performance testing
2013-10-24 8:50 ` Agovic, Sanimir
@ 2013-10-24 14:13 ` Yao Qi
0 siblings, 0 replies; 24+ messages in thread
From: Yao Qi @ 2013-10-24 14:13 UTC (permalink / raw)
To: Agovic, Sanimir; +Cc: gdb-patches
On 10/24/2013 04:49 PM, Agovic, Sanimir wrote:
> + def measure(self, func, id):
> + """Measure the operations done by func with a collection of measurements."""
> + for m in self.measurements:
> + m.start(id)
> +
> + # Enable GC, force GC and disable GC before running test in order to reduce
> + # the interference from GC.
> + gc.enable()
> + gc.collect()
> + gc.disable()
> +
> + func()
> +
> + gc.enable()
> I`d rather exclude gc cycles completely from the measurements or include gc collection
> _after_ you run your perf test. Otherwise the results will depend on the order the perf
> tests are run.
Right, we should disable gc before measurements are started and enable
gc again when measurements are stopped. I'll update the patch.
--
Yao (é½å°§)
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-10-16 9:05 ` Yao Qi
@ 2013-10-24 14:20 ` Yao Qi
2013-10-25 6:33 ` Doug Evans
1 sibling, 0 replies; 24+ messages in thread
From: Yao Qi @ 2013-10-24 14:20 UTC (permalink / raw)
To: gdb-patches
On 10/16/2013 05:04 PM, Yao Qi wrote:
> + def measure(self, func, id):
> + """Measure the operations done by func with a collection of measurements."""
> + for m in self.measurements:
> + m.start(id)
> +
> + # Enable GC, force GC and disable GC before running test in order to reduce
> + # the interference from GC.
> + gc.enable()
> + gc.collect()
> + gc.disable()
> +
> + func()
> +
> + gc.enable()
> +
> + for m in self.measurements:
> + m.stop(id)
In the patch below, method measure is updated to disable gc before
measurements are started and enable gc after measurements are stopped.
--
Yao (é½å°§)
gdb/testsuite/
* lib/perftest.exp: New.
* gdb.perf/lib/perftest/__init__.py: New.
* gdb.perf/lib/perftest/measure.py: New.
* gdb.perf/lib/perftest/perftest.py: New.
* gdb.perf/lib/perftest/reporter.py: New.
* gdb.perf/lib/perftest/testresult.py: New.
---
gdb/testsuite/gdb.perf/lib/perftest/__init__.py | 17 +++
gdb/testsuite/gdb.perf/lib/perftest/measure.py | 146 ++++++++++++++++++++
gdb/testsuite/gdb.perf/lib/perftest/perftest.py | 72 ++++++++++
gdb/testsuite/gdb.perf/lib/perftest/reporter.py | 64 +++++++++
gdb/testsuite/gdb.perf/lib/perftest/testresult.py | 57 ++++++++
gdb/testsuite/lib/perftest.exp | 148 +++++++++++++++++++++
6 files changed, 504 insertions(+), 0 deletions(-)
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
create mode 100644 gdb/testsuite/lib/perftest.exp
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/__init__.py b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
new file mode 100644
index 0000000..7739a3e
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
@@ -0,0 +1,17 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+"""
+GDB performance testing framework.
+"""
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
new file mode 100644
index 0000000..74b643d
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import gc
+
+class Measure(object):
+ """A class that measure and collect the interesting data for a given testcase.
+
+ An instance of Measure has a collection of measurements, and each
+ of them is to measure a given aspect, such as time and memory.
+ """
+
+ def __init__(self, measurements):
+ """Constructor of measure.
+
+ measurements is a collection of Measurement objects.
+ """
+
+ self.measurements = measurements
+
+ def measure(self, func, id):
+ """Measure the operations done by func with a collection of measurements."""
+ # Enable GC, force GC and disable GC before running test in order to reduce
+ # the interference from GC.
+ gc.enable()
+ gc.collect()
+ gc.disable()
+
+ for m in self.measurements:
+ m.start(id)
+
+ func()
+
+ for m in self.measurements:
+ m.stop(id)
+
+ gc.enable()
+
+ def report(self, reporter, name):
+ """Report the measured results."""
+ for m in self.measurements:
+ m.report(reporter, name)
+
+class Measurement(object):
+ """A measurement for a certain aspect."""
+
+ def __init__(self, name, result):
+ """Constructor of Measurement.
+
+ Attribute result is the TestResult associated with measurement.
+ """
+ self.name = name;
+ self.result = result
+
+ def start(self, id):
+ """Abstract method to start the measurement."""
+ raise NotImplementedError("Abstract Method:start")
+
+ def stop(self, id):
+ """Abstract method to stop the measurement.
+
+ When the measurement is stopped, we've got something, and
+ record them in result.
+ """
+ raise NotImplementedError("Abstract Method:stop.")
+
+ def report(self, reporter, name):
+ """Report the measured data by argument reporter."""
+ self.result.report(reporter, name + " " + self.name)
+
+class MeasurementCPUTime(Measurement):
+ """Measurement on CPU time."""
+ # On UNIX, time.clock() measures the amount of CPU time that has
+ # been used by the current process. On Windows it will measure
+ # wall-clock seconds elapsed since the first call to the function.
+ # Something other than time.clock() should be used to measure CPU
+ # time on Windows.
+
+ def __init__(self, result):
+ super(MeasurementCPUTime, self).__init__("cpu_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.clock()
+
+ def stop(self, id):
+ if os.name == 'nt':
+ cpu_time = 0
+ else:
+ cpu_time = time.clock() - self.start_time
+ self.result.record (id, cpu_time)
+
+class MeasurementWallTime(Measurement):
+ """Measurement on Wall time."""
+
+ def __init__(self, result):
+ super(MeasurementWallTime, self).__init__("wall_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.time()
+
+ def stop(self, id):
+ wall_time = time.time() - self.start_time
+ self.result.record (id, wall_time)
+
+class MeasurementVmSize(Measurement):
+ """Measurement on memory usage represented by VmSize."""
+
+ def __init__(self, result):
+ super(MeasurementVmSize, self).__init__("vmsize", result)
+
+ def _compute_process_memory_usage(self, key):
+ file_path = "/proc/%d/status" % os.getpid()
+ try:
+ t = open(file_path)
+ v = t.read()
+ t.close()
+ except:
+ return 0
+ i = v.index(key)
+ v = v[i:].split(None, 3)
+ if len(v) < 3:
+ return 0
+ return int(v[1])
+
+ def start(self, id):
+ pass
+
+ def stop(self, id):
+ memory_used = self._compute_process_memory_usage("VmSize:")
+ self.result.record (id, memory_used)
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/perftest.py b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
new file mode 100644
index 0000000..f7b8a8d
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
@@ -0,0 +1,72 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import testresult
+import reporter
+from measure import Measure
+from measure import MeasurementCPUTime
+from measure import MeasurementWallTime
+from measure import MeasurementVmSize
+
+class TestCase(object):
+ """Base class of all performance testing cases.
+
+ Each sub-class should override methods execute_test, in which
+ several GDB operations are performed and measured by attribute
+ measure. Sub-class can also override method warm_up optionally
+ if the test case needs warm up.
+ """
+
+ def __init__(self, name, measure):
+ """Constructor of TestCase.
+
+ Construct an instance of TestCase with a name and a measure
+ which is to measure the test by several different measurements.
+ """
+
+ self.name = name
+ self.measure = measure
+
+ def execute_test(self):
+ """Abstract method to do the actual tests."""
+ raise NotImplementedError("Abstract Method.")
+
+ def warm_up(self):
+ """Do some operations to warm up the environment."""
+
+ def run(self, warm_up=True, append=True):
+ """Run this test case.
+
+ It is a template method to invoke method warm_up,
+ execute_test, and finally report the measured results.
+ If parameter warm_up is True, run method warm_up. If parameter
+ append is True, the test result will be appended instead of
+ overwritten.
+ """
+ if warm_up:
+ self.warm_up()
+
+ self.execute_test()
+ self.measure.report(reporter.TextReporter(append), self.name)
+
+class TestCaseWithBasicMeasurements(TestCase):
+ """Test case measuring CPU time, wall time and memory usage."""
+
+ def __init__(self, name):
+ result_factory = testresult.SingleStatisticResultFactory()
+ measurements = [MeasurementCPUTime(result_factory.create_result()),
+ MeasurementWallTime(result_factory.create_result()),
+ MeasurementVmSize(result_factory.create_result())]
+ super (TestCaseWithBasicMeasurements, self).__init__ (name, Measure(measurements))
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
new file mode 100644
index 0000000..a902e4b
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
@@ -0,0 +1,64 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class Reporter(object):
+ """Base class of reporter to report test results in a certain format.
+
+ Subclass, which is specific to a report format, should overwrite
+ methods report, start and end.
+ """
+
+ def __init__(self, append):
+ """Constructor of Reporter.
+
+ attribute append is used to determine whether to append or
+ overwrite log file.
+ """
+ self.append = append
+
+ def report(self, *args):
+ raise NotImplementedError("Abstract Method:report.")
+
+ def start(self):
+ """Invoked when reporting is started."""
+ raise NotImplementedError("Abstract Method:start.")
+
+ def end(self):
+ """Invoked when reporting is done.
+
+ It must be overridden to do some cleanups, such as closing file
+ descriptors.
+ """
+ raise NotImplementedError("Abstract Method:end.")
+
+class TextReporter(Reporter):
+ """Report results in a plain text file 'perftest.log'."""
+
+ def __init__(self, append):
+ super (TextReporter, self).__init__(Reporter(append))
+ self.txt_log = None
+
+ def report(self, *args):
+ self.txt_log.write(' '.join(str(arg) for arg in args))
+ self.txt_log.write('\n')
+
+ def start(self):
+ if self.append:
+ self.txt_log = open ("perftest.log", 'a+');
+ else:
+ self.txt_log = open ("perftest.log", 'w');
+
+ def end(self):
+ self.txt_log.close ()
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
new file mode 100644
index 0000000..e571f12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
@@ -0,0 +1,57 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class TestResult(object):
+ """Base class to record and report test results.
+
+ Method record is to record the results of test case, and report
+ method is to report the recorded results by a given reporter.
+ """
+
+ def record(self, parameter, result):
+ raise NotImplementedError("Abstract Method:record.")
+
+ def report(self, reporter, name):
+ """Report the test results by reporter."""
+ raise NotImplementedError("Abstract Method:report.")
+
+class SingleStatisticTestResult(TestResult):
+ """Test results for the test case with a single statistic."""
+
+ def __init__(self):
+ super (SingleStatisticTestResult, self).__init__ ()
+ self.results = dict ()
+
+ def record(self, parameter, result):
+ self.results[parameter] = result
+
+ def report(self, reporter, name):
+ reporter.start()
+ for key in sorted(self.results.iterkeys()):
+ reporter.report(name, key, self.results[key])
+ reporter.end()
+
+class ResultFactory(object):
+ """A factory to create an instance of TestResult."""
+
+ def create_result(self):
+ """Create an instance of TestResult."""
+ raise NotImplementedError("Abstract Method:create_result.")
+
+class SingleStatisticResultFactory(ResultFactory):
+ """A factory to create an instance of SingleStatisticTestResult."""
+
+ def create_result(self):
+ return SingleStatisticTestResult()
diff --git a/gdb/testsuite/lib/perftest.exp b/gdb/testsuite/lib/perftest.exp
new file mode 100644
index 0000000..59b204d
--- /dev/null
+++ b/gdb/testsuite/lib/perftest.exp
@@ -0,0 +1,148 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+namespace eval PerfTest {
+ # The name of python file on build.
+ variable remote_python_file
+
+ # The source files are compiled successfully or not.
+ variable compiled_ok
+
+ # A private method to set up GDB for performance testing.
+ proc _setup_perftest {} {
+ variable remote_python_file
+ global srcdir subdir testfile
+
+ set remote_python_file [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py]
+
+ # Set sys.path for module perftest.
+ gdb_test_no_output "python import os, sys"
+ gdb_test_no_output "python sys.path.insert\(0, os.path.abspath\(\"${srcdir}/${subdir}/lib\"\)\)"
+ gdb_test_no_output "python exec (open ('${remote_python_file}').read ())"
+ }
+
+ # A private method to do some cleanups when performance test is
+ # finished.
+ proc _teardown_perftest {} {
+ variable remote_python_file
+
+ remote_file host delete $remote_python_file
+ }
+
+ # Compile source files of test case. BODY is the tcl code to do
+ # actual compilation and it should invoke 'PerfTest::compiled' if
+ # compilation is successful.
+ proc compile {body} {
+ global GDB_PERFORMANCE
+
+ if { [info exists GDB_PERFORMANCE]
+ && [string compare $GDB_PERFORMANCE "run"] } {
+ variable compiled_ok
+
+ set compiled_ok 0
+ uplevel 2 $body
+
+ if {!$compiled_ok} {
+ untested "Could not compile source files."
+ return -1
+ }
+ }
+ }
+
+ # Mark the compilation is finished successfully.
+ proc compiled {} {
+ variable compiled_ok
+
+ set compiled_ok 1
+ }
+
+ # Start up GDB.
+ proc startup_gdb {body} {
+ variable compiled_ok
+
+ if {!$compiled_ok} {
+ return
+ }
+
+ uplevel 2 $body
+ }
+
+ # Run the performance test.
+ proc run {body} {
+ global timeout
+ global GDB_PERFORMANCE_TIMEOUT
+
+ set oldtimeout $timeout
+ if { [info exists GDB_PERFORMANCE_TIMEOUT] } {
+ set timeout $GDB_PERFORMANCE_TIMEOUT
+ } else {
+ set timeout 3000
+ }
+ uplevel 2 $body
+
+ set timeout $oldtimeout
+ }
+
+ # The top-level interface to PerfTest.
+ # COMPILE is the tcl code to generate and compile source files.
+ # STARTUP is the tcl code to start up GDB.
+ # RUN is the tcl code to drive GDB to do some operations.
+ proc assemble {compile startup run} {
+ variable compiled_ok
+ global GDB_PERFORMANCE
+
+ set compiled_ok 1
+ eval $compile
+
+ if {!$compiled_ok} {
+ return
+ }
+
+ # Don't execute the run if GDB_PERFORMANCE=compile.
+ if { [info exists GDB_PERFORMANCE]
+ && [string compare $GDB_PERFORMANCE "compile"] == 0} {
+ return
+ }
+
+ eval $startup
+
+ _setup_perftest
+
+ eval $run
+
+ _teardown_perftest
+ }
+}
+
+# Return true if performance tests are skipped.
+
+proc skip_perf_tests { } {
+ global GDB_PERFORMANCE
+
+ if [info exists GDB_PERFORMANCE] {
+
+ if { [string compare $GDB_PERFORMANCE "compile"]
+ && [string compare $GDB_PERFORMANCE "run"]
+ && [string compare $GDB_PERFORMANCE "both"] } {
+ # GDB_PERFORMANCE=compile|run|both is allowed.
+ unsupported "Unknown value of GDB_PERFORMANCE."
+ return 1
+ }
+
+ return 0
+ }
+
+ return 1
+}
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 0/4 V3] GDB Performance testing
2013-10-16 7:10 [PATCH 0/4 V3] GDB Performance testing Yao Qi
` (4 preceding siblings ...)
2013-10-24 2:44 ` ping: [PATCH 0/4 V3] GDB Performance testing Yao Qi
@ 2013-10-25 2:59 ` Doug Evans
5 siblings, 0 replies; 24+ messages in thread
From: Doug Evans @ 2013-10-25 2:59 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
On Wed, Oct 16, 2013 at 12:09 AM, Yao Qi <yao@codesourcery.com> wrote:
> Hi,
> Here is the V3 of GDB performance testing. The changes in V3 can be
> found in each patch, and they address all review comments.
Cool.
> $ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver GDB_PERFORMANCE_TIMEOUT=4000 solib.exp"
I haven't read the rest in the series yet, but while it's on my mind I
want to write this down so I don't forget.
Another thing the test harness needs is the ability to set a limit on
the memory size of gdb.
It's a safety net because sometimes changes happen that cause memory
usage to really swell and we don't want to drag a machine into
thrashing just to get through the test.
Using system monitored settings for both cpu and memory usage would be
good (ulimit).
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 1/4] New make target 'check-perf' and new dir gdb.perf
2013-10-16 7:10 ` [PATCH 1/4] New make target 'check-perf' and new dir gdb.perf Yao Qi
@ 2013-10-25 3:04 ` Doug Evans
2013-10-25 17:34 ` Pedro Alves
0 siblings, 1 reply; 24+ messages in thread
From: Doug Evans @ 2013-10-25 3:04 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
On Wed, Oct 16, 2013 at 12:09 AM, Yao Qi <yao@codesourcery.com> wrote:
> We add a new dir gdb.perf in testsuite for all performance tests.
> However, current 'make check' logic will either run dejagnu in
> directory testsuite or iterate all gdb.* directories which has *.exp
> files. Both of them will run tests in gdb.perf. We want to achieve:
>
> 1) typical 'make check' should not run performance tests. In each perf
> test case, GDB_PERFORMANCE is checked. If it doesn't exist, return.
> 2) run perf tests easily. We add a new makefile target 'check-perf'.
>
> V2 is simpler than V1, since we don't have to filter out gdb.perf
> directory.
>
> V3: Move GDB_PERFORMANCE=both appears before RUNTESTFLAGS. It was
> approved by Doug.
>
> gdb:
>
> 2013-10-16 Yao Qi <yao@codesourcery.com>
>
> * Makefile.in (check-perf): New target.
>
> gdb/testsuite:
>
> 2013-10-16 Yao Qi <yao@codesourcery.com>
>
> * Makefile.in (check-perf): New target.
> * configure.ac (AC_OUTPUT): Output Makefile in gdb.perf.
> * configure: Re-generated.
> * gdb.perf/Makefile.in: New.
No more comments to add here, other than while I think I suggested
"GDB_PERFORMANCE=both" every time I read it I keep thinking there has
to be a better word to use here than "both". :-)
"compile-and-run" is too long.
I'm not suggesting changing anything, just wondering if anyone else
has a better word.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-10-16 7:10 ` [PATCH 2/4] Perf test framework Yao Qi
2013-10-16 9:05 ` Yao Qi
@ 2013-10-25 6:32 ` Doug Evans
2013-10-25 8:57 ` Yao Qi
1 sibling, 1 reply; 24+ messages in thread
From: Doug Evans @ 2013-10-25 6:32 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
Hi. Just a few nits inline.
I need to download the patch and play with it!
On Wed, Oct 16, 2013 at 12:09 AM, Yao Qi <yao@codesourcery.com> wrote:
> This patch adds a basic framework to do performance testing for GDB.
> perftest.py is about the test case, testresult.py is about test
> results, and how are they saved. reporter.py is about how results
> are reported (in what format). measure.py is about measuring the
> execution of tests by a collection of measurements.
>
> In V3, there are some changes,
>
> - Add wall time measurement, cpu time measurement and vmsize
> measurement.
> - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements,
> which measures cpu time, wall time, and memory (vmsize).
> - GDB_PERFORMANCE=run|compile|both to control the mode of perf
> testing.
> - New GDB_PERFORMANCE_TIMEOUT to specify the timeout.
> - Split proc prepare to proc compile and startup.
> - Disable GC while doing measurements.
>
> In V2, there are several changes to address Doug and Sanimir's
> comments.
>
> - Add copyright header and docstring in perftest/__init__.py
> - Remove config.py.
> - Fix docstring format.
> - Rename classes "SingleVariable" to "SingleStatistic".
> - Don't extend gdb.Function in class TestCase. Add a new method run
> to run the test case so that we can pass parameters to test.
> - Allow to customize whether to warm up and to append test log.
> - Move time measurement into test harness. Add a new class
> Measurement for a specific measurement and a new class Measure to
> measure them for a given test case.
> - A new class ResultFactory to create instances of TestResult.
> - New file lib/perftest.exp, which is to do some preparations and
> cleanups to simplify each *.exp file.
> - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set.
>
> gdb/testsuite/
>
> * lib/perftest.exp: New.
> * gdb.perf/lib/perftest/__init__.py: New.
> * gdb.perf/lib/perftest/measure.py: New.
> * gdb.perf/lib/perftest/perftest.py: New.
> * gdb.perf/lib/perftest/reporter.py: New.
> * gdb.perf/lib/perftest/testresult.py: New.
> ---
> gdb/testsuite/gdb.perf/lib/perftest/__init__.py | 17 +++
> gdb/testsuite/gdb.perf/lib/perftest/measure.py | 146 ++++++++++++++++++++
> gdb/testsuite/gdb.perf/lib/perftest/perftest.py | 72 ++++++++++
> gdb/testsuite/gdb.perf/lib/perftest/reporter.py | 64 +++++++++
> gdb/testsuite/gdb.perf/lib/perftest/testresult.py | 57 ++++++++
> gdb/testsuite/lib/perftest.exp | 148 +++++++++++++++++++++
> 6 files changed, 504 insertions(+), 0 deletions(-)
> create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
> create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
> create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
> create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
> create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
> create mode 100644 gdb/testsuite/lib/perftest.exp
>
> diff --git a/gdb/testsuite/gdb.perf/lib/perftest/__init__.py b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
> new file mode 100644
> index 0000000..7739a3e
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
> @@ -0,0 +1,17 @@
> +# Copyright (C) 2013 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +"""
> +GDB performance testing framework.
> +"""
> diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
> new file mode 100644
> index 0000000..7478be5
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
> @@ -0,0 +1,146 @@
> +# Copyright (C) 2013 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +import time
> +import os
> +import gc
> +
> +class Measure(object):
> + """A class that measure and collect the interesting data for a given testcase.
> +
> + An instance of Measure has a collection of measurements, and each
> + of them is to measure a given aspect, such as time and memory.
> + """
> +
> + def __init__(self, measurements):
> + """Constructor of measure.
> +
> + measurements is a collection of Measurement objects.
> + """
> +
> + self.measurements = measurements
> +
> + def measure(self, func, id):
> + """Measure the operations done by func with a collection of measurements."""
> + for m in self.measurements:
> + m.start(id)
> +
> + # Enable GC, force GC and disable GC before running test in order to reduce
> + # the interference from GC.
> + gc.enable()
> + gc.collect()
> + gc.disable()
> +
> + func()
> +
> + gc.enable()
> +
> + for m in self.measurements:
> + m.stop(id)
> +
> + def report(self, reporter, name):
> + """Report the measured results."""
> + for m in self.measurements:
> + m.report(reporter, name)
> +
> +class Measurement(object):
> + """A measurement for a certain aspect."""
> +
> + def __init__(self, name, result):
> + """Constructor of Measurement.
> +
> + Attribute result is the TestResult associated with measurement.
> + """
> + self.name = name;
> + self.result = result
> +
> + def start(self, id):
> + """Abstract method to start the measurement."""
> + raise NotImplementedError("Abstract Method:start")
> +
> + def stop(self, id):
> + """Abstract method to stop the measurement.
> +
> + When the measurement is stopped, we've got something, and
> + record them in result.
> + """
> + raise NotImplementedError("Abstract Method:stop.")
> +
> + def report(self, reporter, name):
> + """Report the measured data by argument reporter."""
> + self.result.report(reporter, name + " " + self.name)
> +
> +class MeasurementCPUTime(Measurement):
> + """Measurement on CPU time."""
> + # On UNIX, time.clock() measures the amount of CPU time that has
> + # been used by the current process. On Windows it will measure
> + # wall-clock seconds elapsed since the first call to the function.
> + # Something other than time.clock() should be used to measure CPU
> + # time on Windows.
> +
> + def __init__(self, result):
> + super(MeasurementCPUTime, self).__init__("cpu_time", result)
> + self.start_time = 0
> +
> + def start(self, id):
> + self.start_time = time.clock()
> +
> + def stop(self, id):
> + if os.name == 'nt':
> + cpu_time = 0
> + else:
> + cpu_time = time.clock() - self.start_time
> + self.result.record (id, cpu_time)
> +
> +class MeasurementWallTime(Measurement):
> + """Measurement on Wall time."""
> +
> + def __init__(self, result):
> + super(MeasurementWallTime, self).__init__("wall_time", result)
> + self.start_time = 0
> +
> + def start(self, id):
> + self.start_time = time.time()
> +
> + def stop(self, id):
> + wall_time = time.time() - self.start_time
> + self.result.record (id, wall_time)
> +
> +class MeasurementVmSize(Measurement):
> + """Measurement on memory usage represented by VmSize."""
> +
> + def __init__(self, result):
> + super(MeasurementVmSize, self).__init__("vmsize", result)
> +
> + def _compute_process_memory_usage(self, key):
> + file_path = "/proc/%d/status" % os.getpid()
> + try:
> + t = open(file_path)
> + v = t.read()
> + t.close()
> + except:
> + return 0
> + i = v.index(key)
> + v = v[i:].split(None, 3)
> + if len(v) < 3:
> + return 0
> + return int(v[1])
> +
> + def start(self, id):
> + foo = 0
Replace "foo = 0" with "pass".
I think that's the preferred way to do this.
> +
> + def stop(self, id):
> + memory_used = self._compute_process_memory_usage("VmSize:")
> + self.result.record (id, memory_used)
> diff --git a/gdb/testsuite/gdb.perf/lib/perftest/perftest.py b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
> new file mode 100644
> index 0000000..f7b8a8d
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
> @@ -0,0 +1,72 @@
> +# Copyright (C) 2013 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +import testresult
> +import reporter
> +from measure import Measure
> +from measure import MeasurementCPUTime
> +from measure import MeasurementWallTime
> +from measure import MeasurementVmSize
I don't have a strong preference, but it's odd the "CPU" is all caps
and "Vm" isn't.
> +
> +class TestCase(object):
> + """Base class of all performance testing cases.
> +
> + Each sub-class should override methods execute_test, in which
> + several GDB operations are performed and measured by attribute
> + measure. Sub-class can also override method warm_up optionally
> + if the test case needs warm up.
> + """
> +
> + def __init__(self, name, measure):
> + """Constructor of TestCase.
> +
> + Construct an instance of TestCase with a name and a measure
> + which is to measure the test by several different measurements.
> + """
> +
> + self.name = name
> + self.measure = measure
> +
> + def execute_test(self):
> + """Abstract method to do the actual tests."""
> + raise NotImplementedError("Abstract Method.")
> +
> + def warm_up(self):
> + """Do some operations to warm up the environment."""
I"m not sure what the conventions are for an empty function with a doc string.
Do we need to add "pass" here?
> +
> + def run(self, warm_up=True, append=True):
> + """Run this test case.
> +
> + It is a template method to invoke method warm_up,
> + execute_test, and finally report the measured results.
> + If parameter warm_up is True, run method warm_up. If parameter
> + append is True, the test result will be appended instead of
> + overwritten.
> + """
> + if warm_up:
> + self.warm_up()
> +
> + self.execute_test()
> + self.measure.report(reporter.TextReporter(append), self.name)
> +
> +class TestCaseWithBasicMeasurements(TestCase):
> + """Test case measuring CPU time, wall time and memory usage."""
> +
> + def __init__(self, name):
> + result_factory = testresult.SingleStatisticResultFactory()
> + measurements = [MeasurementCPUTime(result_factory.create_result()),
> + MeasurementWallTime(result_factory.create_result()),
> + MeasurementVmSize(result_factory.create_result())]
> + super (TestCaseWithBasicMeasurements, self).__init__ (name, Measure(measurements))
> diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
> new file mode 100644
> index 0000000..a902e4b
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
> @@ -0,0 +1,64 @@
> +# Copyright (C) 2013 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +class Reporter(object):
> + """Base class of reporter to report test results in a certain format.
> +
> + Subclass, which is specific to a report format, should overwrite
> + methods report, start and end.
> + """
> +
> + def __init__(self, append):
> + """Constructor of Reporter.
> +
> + attribute append is used to determine whether to append or
> + overwrite log file.
> + """
> + self.append = append
> +
> + def report(self, *args):
> + raise NotImplementedError("Abstract Method:report.")
> +
> + def start(self):
> + """Invoked when reporting is started."""
> + raise NotImplementedError("Abstract Method:start.")
> +
> + def end(self):
> + """Invoked when reporting is done.
> +
> + It must be overridden to do some cleanups, such as closing file
> + descriptors.
> + """
> + raise NotImplementedError("Abstract Method:end.")
> +
> +class TextReporter(Reporter):
> + """Report results in a plain text file 'perftest.log'."""
> +
> + def __init__(self, append):
> + super (TextReporter, self).__init__(Reporter(append))
> + self.txt_log = None
> +
> + def report(self, *args):
> + self.txt_log.write(' '.join(str(arg) for arg in args))
> + self.txt_log.write('\n')
> +
> + def start(self):
> + if self.append:
> + self.txt_log = open ("perftest.log", 'a+');
> + else:
> + self.txt_log = open ("perftest.log", 'w');
> +
> + def end(self):
> + self.txt_log.close ()
> diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
> new file mode 100644
> index 0000000..e571f12
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
> @@ -0,0 +1,57 @@
> +# Copyright (C) 2013 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +class TestResult(object):
> + """Base class to record and report test results.
> +
> + Method record is to record the results of test case, and report
> + method is to report the recorded results by a given reporter.
> + """
> +
> + def record(self, parameter, result):
> + raise NotImplementedError("Abstract Method:record.")
> +
> + def report(self, reporter, name):
> + """Report the test results by reporter."""
> + raise NotImplementedError("Abstract Method:report.")
> +
> +class SingleStatisticTestResult(TestResult):
> + """Test results for the test case with a single statistic."""
> +
> + def __init__(self):
> + super (SingleStatisticTestResult, self).__init__ ()
> + self.results = dict ()
> +
> + def record(self, parameter, result):
> + self.results[parameter] = result
> +
> + def report(self, reporter, name):
> + reporter.start()
> + for key in sorted(self.results.iterkeys()):
> + reporter.report(name, key, self.results[key])
> + reporter.end()
> +
> +class ResultFactory(object):
> + """A factory to create an instance of TestResult."""
> +
> + def create_result(self):
> + """Create an instance of TestResult."""
> + raise NotImplementedError("Abstract Method:create_result.")
> +
> +class SingleStatisticResultFactory(ResultFactory):
> + """A factory to create an instance of SingleStatisticTestResult."""
> +
> + def create_result(self):
> + return SingleStatisticTestResult()
> diff --git a/gdb/testsuite/lib/perftest.exp b/gdb/testsuite/lib/perftest.exp
> new file mode 100644
> index 0000000..59b204d
> --- /dev/null
> +++ b/gdb/testsuite/lib/perftest.exp
> @@ -0,0 +1,148 @@
> +# Copyright (C) 2013 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +namespace eval PerfTest {
> + # The name of python file on build.
> + variable remote_python_file
> +
> + # The source files are compiled successfully or not.
> + variable compiled_ok
> +
> + # A private method to set up GDB for performance testing.
> + proc _setup_perftest {} {
> + variable remote_python_file
> + global srcdir subdir testfile
> +
> + set remote_python_file [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py]
> +
> + # Set sys.path for module perftest.
> + gdb_test_no_output "python import os, sys"
> + gdb_test_no_output "python sys.path.insert\(0, os.path.abspath\(\"${srcdir}/${subdir}/lib\"\)\)"
> + gdb_test_no_output "python exec (open ('${remote_python_file}').read ())"
> + }
> +
> + # A private method to do some cleanups when performance test is
> + # finished.
> + proc _teardown_perftest {} {
> + variable remote_python_file
> +
> + remote_file host delete $remote_python_file
> + }
> +
> + # Compile source files of test case. BODY is the tcl code to do
> + # actual compilation and it should invoke 'PerfTest::compiled' if
> + # compilation is successful.
> + proc compile {body} {
> + global GDB_PERFORMANCE
> +
> + if { [info exists GDB_PERFORMANCE]
> + && [string compare $GDB_PERFORMANCE "run"] } {
> + variable compiled_ok
> +
> + set compiled_ok 0
> + uplevel 2 $body
> +
> + if {!$compiled_ok} {
> + untested "Could not compile source files."
> + return -1
> + }
> + }
> + }
> +
> + # Mark the compilation is finished successfully.
> + proc compiled {} {
> + variable compiled_ok
> +
> + set compiled_ok 1
> + }
> +
> + # Start up GDB.
> + proc startup_gdb {body} {
> + variable compiled_ok
> +
> + if {!$compiled_ok} {
> + return
> + }
> +
> + uplevel 2 $body
> + }
> +
> + # Run the performance test.
> + proc run {body} {
> + global timeout
> + global GDB_PERFORMANCE_TIMEOUT
> +
> + set oldtimeout $timeout
> + if { [info exists GDB_PERFORMANCE_TIMEOUT] } {
> + set timeout $GDB_PERFORMANCE_TIMEOUT
> + } else {
> + set timeout 3000
> + }
> + uplevel 2 $body
> +
> + set timeout $oldtimeout
> + }
> +
> + # The top-level interface to PerfTest.
> + # COMPILE is the tcl code to generate and compile source files.
> + # STARTUP is the tcl code to start up GDB.
> + # RUN is the tcl code to drive GDB to do some operations.
> + proc assemble {compile startup run} {
> + variable compiled_ok
> + global GDB_PERFORMANCE
> +
> + set compiled_ok 1
> + eval $compile
> +
> + if {!$compiled_ok} {
> + return
> + }
> +
> + # Don't execute the run if GDB_PERFORMANCE=compile.
> + if { [info exists GDB_PERFORMANCE]
> + && [string compare $GDB_PERFORMANCE "compile"] == 0} {
> + return
> + }
> +
> + eval $startup
> +
> + _setup_perftest
> +
> + eval $run
> +
> + _teardown_perftest
> + }
> +}
> +
> +# Return true if performance tests are skipped.
> +
> +proc skip_perf_tests { } {
> + global GDB_PERFORMANCE
> +
> + if [info exists GDB_PERFORMANCE] {
> +
> + if { [string compare $GDB_PERFORMANCE "compile"]
> + && [string compare $GDB_PERFORMANCE "run"]
> + && [string compare $GDB_PERFORMANCE "both"] } {
How about just "$GDB_PERFORMANCE" != "compile", etc. ?
> + # GDB_PERFORMANCE=compile|run|both is allowed.
> + unsupported "Unknown value of GDB_PERFORMANCE."
> + return 1
> + }
> +
> + return 0
> + }
> +
> + return 1
> +}
> --
> 1.7.7.6
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-10-16 9:05 ` Yao Qi
2013-10-24 14:20 ` Yao Qi
@ 2013-10-25 6:33 ` Doug Evans
1 sibling, 0 replies; 24+ messages in thread
From: Doug Evans @ 2013-10-25 6:33 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
On Wed, Oct 16, 2013 at 2:04 AM, Yao Qi <yao@codesourcery.com> wrote:
> On 10/16/2013 03:09 PM, Yao Qi wrote:
>> + def start(self, id):
>> + foo = 0
>
> Missed the change during git rebase. "pass" should be used here.
> Here is an updated one.
Heh. Read this after sending my review.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 3/4] Mention perf test in testsuite/README
2013-10-16 7:10 ` [PATCH 3/4] Mention perf test in testsuite/README Yao Qi
@ 2013-10-25 6:43 ` Doug Evans
0 siblings, 0 replies; 24+ messages in thread
From: Doug Evans @ 2013-10-25 6:43 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
On Wed, Oct 16, 2013 at 12:09 AM, Yao Qi <yao@codesourcery.com> wrote:
> gdb/testsuite:
>
> 2013-10-16 Yao Qi <yao@codesourcery.com>
>
> * README: Mention performance tests.
Ok by me.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 4/4] Test on solib load and unload
2013-10-16 7:10 ` [PATCH 4/4] Test on solib load and unload Yao Qi
@ 2013-10-25 7:17 ` Doug Evans
0 siblings, 0 replies; 24+ messages in thread
From: Doug Evans @ 2013-10-25 7:17 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
Hi.
A couple of nit, otherwise seems ok to me.
On Wed, Oct 16, 2013 at 12:09 AM, Yao Qi <yao@codesourcery.com> wrote:
> This patch is to add a test case to on the performance of GDB handling
> load and unload of shared library.
>
> In V3, there are some changes,
>
> - Adapt to perf test framework changes.
> - Measure load and unload separately.
>
> In V2, there are some changes,
>
> - A new proc gdb_produce_source to produce source files. I tried to
> move all source file generation code out of solib.exp, but
> compilation step still needs to know the generated file names. I
> have to hard-code the file names in compilation step, which is not
> good to me, so I give up on this moving.
> - SOLIB_NUMBER -> SOLIB_COUNT
> - New variable SOLIB_DLCLOSE_REVERSED_ORDER to control the order of
> iterating a list of shared libs to dlclose them.
> - New variable GDB_PERFORMANCE to enable these perf test cases.
> - Remove dlsym call in solib.c.
> - Update solib.py for the updated framework.
>
> gdb/testsuite/
>
> * lib/gdb.exp (gdb_produce_source): New procedure.
> * gdb.perf/solib.c: New.
> * gdb.perf/solib.exp: New.
> * gdb.perf/solib.py: New.
> ---
> gdb/testsuite/gdb.perf/solib.c | 78 ++++++++++++++++++++++++++++++++++++++
> gdb/testsuite/gdb.perf/solib.exp | 78 ++++++++++++++++++++++++++++++++++++++
> gdb/testsuite/gdb.perf/solib.py | 73 +++++++++++++++++++++++++++++++++++
> gdb/testsuite/lib/gdb.exp | 16 ++++++++
> 4 files changed, 245 insertions(+), 0 deletions(-)
> create mode 100644 gdb/testsuite/gdb.perf/solib.c
> create mode 100644 gdb/testsuite/gdb.perf/solib.exp
> create mode 100644 gdb/testsuite/gdb.perf/solib.py
>
> diff --git a/gdb/testsuite/gdb.perf/solib.c b/gdb/testsuite/gdb.perf/solib.c
> new file mode 100644
> index 0000000..42973d6
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/solib.c
> @@ -0,0 +1,78 @@
> +/* This testcase is part of GDB, the GNU debugger.
> +
> + Copyright (C) 2013 Free Software Foundation, Inc.
> +
> + This program is free software; you can redistribute it and/or modify
> + it under the terms of the GNU General Public License as published by
> + the Free Software Foundation; either version 3 of the License, or
> + (at your option) any later version.
> +
> + This program is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + GNU General Public License for more details.
> +
> + You should have received a copy of the GNU General Public License
> + along with this program. If not, see <http://www.gnu.org/licenses/>. */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +
> +#ifdef __WIN32__
> +#include <windows.h>
> +#define dlopen(name, mode) LoadLibrary (TEXT (name))
> +# define dlsym(handle, func) GetProcAddress (handle, func)
> +#define dlclose(handle) FreeLibrary (handle)
> +#else
> +#include <dlfcn.h>
> +#endif
> +
> +static void **handles;
> +
> +void
> +do_test_load (int number)
> +{
> + char libname[40];
> + int i;
> +
> + handles = malloc (sizeof (void *) * number);
Check malloc result?
> +
> + for (i = 0; i < number; i++)
> + {
> + sprintf (libname, "solib-lib%d", i);
> + handles[i] = dlopen (libname, RTLD_LAZY);
> + if (handles[i] == NULL)
> + {
> + printf ("ERROR on dlopen %s\n", libname);
> + return;
> + }
> + }
> +}
> +
> +void
> +do_test_unload (int number)
> +{
> + int i;
> +
> + /* Unload shared libraries in different orders. */
> +#ifndef SOLIB_DLCLOSE_REVERSED_ORDER
> + for (i = 0; i < number; i++)
> +#else
> + for (i = number - 1; i >= 0; i--)
> +#endif
> + dlclose (handles[i]);
> +
> + free (handles);
> +}
> +
> +static void
> +end (void)
> +{}
> +
> +int
> +main (void)
> +{
> + end ();
> +
> + return 0;
> +}
> diff --git a/gdb/testsuite/gdb.perf/solib.exp b/gdb/testsuite/gdb.perf/solib.exp
> new file mode 100644
> index 0000000..fc66f77
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/solib.exp
> @@ -0,0 +1,78 @@
> +# Copyright (C) 2013 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +# This test case is to test the performance of GDB when it is handling
> +# the shared libraries of inferior are loaded and unloaded.
> +load_lib perftest.exp
> +
> +if [skip_perf_tests] {
> + return 0
> +}
> +
> +standard_testfile .c
> +set executable $testfile
> +set expfile $testfile.exp
> +
> +# make check-perf RUNTESTFLAGS='solib.exp SOLIB_COUNT=1024'
> +if ![info exists SOLIB_COUNT] {
> + set SOLIB_COUNT 128
> +}
> +
> +PerfTest::assemble {
> + compile {
> + for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
> +
> + # Produce source files.
> + set libname "solib-lib$i"
> + set src [standard_output_file $libname.c]
> + set exe [standard_output_file $libname]
> +
> + gdb_produce_source $src { "int shr$i (void) {return 0;}" }
> +
> + # Compile.
> + if { [gdb_compile_shlib $src $exe {debug}] != "" } {
> + untested "Couldn't compile $src."
> + return -1
> + }
> +
> + # Delete object files to save some space.
> + file delete [standard_output_file "solib-lib$i.c.o"]
> + }
> +
> + set compile_flags {debug shlib_load}
> + global SOLIB_DLCLOSE_REVERSED_ORDER
IWBN to have a convention to document all the test parameters in one
place, e.g. at the top of the .exp file or some such.
SOLIB_DLCLOSE_REVERSED_ORDER is kinda buried in the test.
> +
> + if [info exists SOLIB_DLCLOSE_REVERSED_ORDER] {
> + lappend compile_flags "additional_flags=-DSOLIB_DLCLOSE_REVERSED_ORDER"
> + }
> +
> + if { [gdb_compile "$srcdir/$subdir/$srcfile" ${binfile} executable $compile_flags] == "" } {
> + PerfTest::compiled
> + }
> + }
> +} {
> + startup_gdb {
> + clean_restart $binfile
> +
> + if ![runto_main] {
> + fail "Can't run to main"
> + return -1
> + }
> + }
> +} {
> + run {
> + gdb_test_no_output "python SolibLoadUnload\($SOLIB_COUNT\).run()"
> + }
> +}
> diff --git a/gdb/testsuite/gdb.perf/solib.py b/gdb/testsuite/gdb.perf/solib.py
> new file mode 100644
> index 0000000..91393b1
> --- /dev/null
> +++ b/gdb/testsuite/gdb.perf/solib.py
> @@ -0,0 +1,73 @@
> +# Copyright (C) 2013 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +# This test case is to test the speed of GDB when it is handling the
> +# shared libraries of inferior are loaded and unloaded.
> +
> +from perftest import perftest
> +from perftest import measure
> +
> +class SolibLoadUnload1(perftest.TestCaseWithBasicMeasurements):
> + def __init__(self, solib_count, measure_load):
> + if measure_load:
> + name = "solib_load"
> + else:
> + name = "solib_unload"
> + # We want to measure time in this test.
> + super (SolibLoadUnload1, self).__init__ (name)
> + self.solib_count = solib_count
> + self.measure_load = measure_load
> +
> + def warm_up(self):
> + do_test_load = "call do_test_load (%d)" % self.solib_count
> + do_test_unload = "call do_test_unload (%d)" % self.solib_count
> + gdb.execute(do_test_load)
> + gdb.execute(do_test_unload)
> +
> + def execute_test(self):
> + num = self.solib_count
> + iteration = 5;
> +
> + while num > 0 and iteration > 0:
> + # Do inferior calls to do_test_load and do_test_unload in pairs,
> + # but measure differently.
> + if self.measure_load:
> + do_test_load = "call do_test_load (%d)" % num
> + func = lambda: gdb.execute (do_test_load)
> +
> + self.measure.measure(func, num)
> +
> + do_test_unload = "call do_test_unload (%d)" % num
> + gdb.execute (do_test_unload)
> +
> + else:
> + do_test_load = "call do_test_load (%d)" % num
> + gdb.execute (do_test_load)
> +
> + do_test_unload = "call do_test_unload (%d)" % num
> + func = lambda: gdb.execute (do_test_unload)
> +
> + self.measure.measure(func, num)
> +
> + num = num / 2
> + iteration -= 1
> +
> +class SolibLoadUnload(object):
> + def __init__(self, solib_count):
> + self.solib_count = solib_count;
> +
> + def run(self):
> + SolibLoadUnload1(self.solib_count, True).run()
> + SolibLoadUnload1(self.solib_count, False).run()
> diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
> index 3efd539..28aca53 100644
> --- a/gdb/testsuite/lib/gdb.exp
> +++ b/gdb/testsuite/lib/gdb.exp
> @@ -1796,6 +1796,22 @@ proc supports_reverse {} {
> return 0
> }
>
> +# Produce source file NAME and write SOURCES into it.
> +
> +proc gdb_produce_source { name sources } {
> + set index 0
> + set f [open $name "w"]
> +
> + while { ${index} < [llength ${sources}] } {
> + set line [lindex ${sources} ${index}]
> + set index [expr ${index} + 1]
> +
> + set line [uplevel list $line]
> + puts $f $line
> + }
> + close $f
> +}
> +
> # Return 1 if target is ILP32.
> # This cannot be decided simply from looking at the target string,
> # as it might depend on externally passed compiler options like -m64.
> --
> 1.7.7.6
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-10-25 6:32 ` Doug Evans
@ 2013-10-25 8:57 ` Yao Qi
0 siblings, 0 replies; 24+ messages in thread
From: Yao Qi @ 2013-10-25 8:57 UTC (permalink / raw)
To: Doug Evans; +Cc: gdb-patches
On 10/25/2013 02:32 PM, Doug Evans wrote:
> Hi. Just a few nits inline.
> I need to download the patch and play with it!
You can fetch them from
https://github.com/qiyao/gdb/tree/gdb-perf-test-v3_1 . This tree
includes updates to address comments to V3 and it also includes extra
three test cases on backtrace, single step and disassemble.
--
Yao (é½å°§)
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 1/4] New make target 'check-perf' and new dir gdb.perf
2013-10-25 3:04 ` Doug Evans
@ 2013-10-25 17:34 ` Pedro Alves
0 siblings, 0 replies; 24+ messages in thread
From: Pedro Alves @ 2013-10-25 17:34 UTC (permalink / raw)
To: Doug Evans; +Cc: Yao Qi, gdb-patches
On 10/25/2013 04:04 AM, Doug Evans wrote:
> On Wed, Oct 16, 2013 at 12:09 AM, Yao Qi <yao@codesourcery.com> wrote:
>> We add a new dir gdb.perf in testsuite for all performance tests.
>> However, current 'make check' logic will either run dejagnu in
>> directory testsuite or iterate all gdb.* directories which has *.exp
>> files. Both of them will run tests in gdb.perf. We want to achieve:
>>
>> 1) typical 'make check' should not run performance tests. In each perf
>> test case, GDB_PERFORMANCE is checked. If it doesn't exist, return.
>> 2) run perf tests easily. We add a new makefile target 'check-perf'.
>>
>> V2 is simpler than V1, since we don't have to filter out gdb.perf
>> directory.
>>
>> V3: Move GDB_PERFORMANCE=both appears before RUNTESTFLAGS. It was
>> approved by Doug.
>>
>> gdb:
>>
>> 2013-10-16 Yao Qi <yao@codesourcery.com>
>>
>> * Makefile.in (check-perf): New target.
>>
>> gdb/testsuite:
>>
>> 2013-10-16 Yao Qi <yao@codesourcery.com>
>>
>> * Makefile.in (check-perf): New target.
>> * configure.ac (AC_OUTPUT): Output Makefile in gdb.perf.
>> * configure: Re-generated.
>> * gdb.perf/Makefile.in: New.
>
> No more comments to add here, other than while I think I suggested
> "GDB_PERFORMANCE=both" every time I read it I keep thinking there has
> to be a better word to use here than "both". :-)
> "compile-and-run" is too long.
> I'm not suggesting changing anything, just wondering if anyone else
> has a better word.
"all" as in "do all steps" comes to mind. If we ever add more
steps, then the meaning of "all" won't change. OTOH, "both" would
get outdated and ambiguous then.
But I think my preference would be to hide GDB_PERFORMANCE as an
internal implementation detail, and instead have extra
make targets:
$ make check-perf-compile // just compile
$ make check-perf-run // just run
$ make check-perf // do all steps
Fewer letters to type, even. Sorry if that was suggested
before and I missed it.
(I'd rename GDB_PERFORMANCE to GDB_PERFTEST_MODE, and
GDB_PERFORMANCE_TIMEOUT->GDB_PERFTEST_TIMEOUT while at it.)
--
Pedro Alves
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-11-04 16:27 ` Tom Tromey
@ 2013-11-06 5:30 ` Yao Qi
0 siblings, 0 replies; 24+ messages in thread
From: Yao Qi @ 2013-11-06 5:30 UTC (permalink / raw)
To: Tom Tromey; +Cc: gdb-patches
On 11/05/2013 12:24 AM, Tom Tromey wrote:
> Yao> + if { "$GDB_PERFTEST_MODE" != "compile"
> Yao> + && "$GDB_PERFTEST_MODE" != "run"
> Yao> + && "$GDB_PERFTEST_MODE" != "both" } {
> Yao> + # GDB_PERFTEST_MODE=compile|run|both is allowed.
> Yao> + unsupported "Unknown value of GDB_PERFTEST_MODE."
> Yao> + return 1
>
> This is basically an internal error.
> Calling "error" and having a backtrace in the log seems better to me.
Fixed it in the commit. Thanks for the review.
--
Yao (é½å°§)
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-11-02 2:42 ` Yao Qi
@ 2013-11-04 16:27 ` Tom Tromey
2013-11-06 5:30 ` Yao Qi
0 siblings, 1 reply; 24+ messages in thread
From: Tom Tromey @ 2013-11-04 16:27 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
>>>>> "Yao" == Yao Qi <yao@codesourcery.com> writes:
Yao> * lib/perftest.exp: New.
Yao> * gdb.perf/lib/perftest/__init__.py: New.
Yao> * gdb.perf/lib/perftest/measure.py: New.
Yao> * gdb.perf/lib/perftest/perftest.py: New.
Yao> * gdb.perf/lib/perftest/reporter.py: New.
Yao> * gdb.perf/lib/perftest/testresult.py: New.
Thanks for making that change, Yao.
One more little nit:
Yao> + if { "$GDB_PERFTEST_MODE" != "compile"
Yao> + && "$GDB_PERFTEST_MODE" != "run"
Yao> + && "$GDB_PERFTEST_MODE" != "both" } {
Yao> + # GDB_PERFTEST_MODE=compile|run|both is allowed.
Yao> + unsupported "Unknown value of GDB_PERFTEST_MODE."
Yao> + return 1
This is basically an internal error.
Calling "error" and having a backtrace in the log seems better to me.
Tom
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-11-01 20:46 ` Tom Tromey
@ 2013-11-02 2:42 ` Yao Qi
2013-11-04 16:27 ` Tom Tromey
0 siblings, 1 reply; 24+ messages in thread
From: Yao Qi @ 2013-11-02 2:42 UTC (permalink / raw)
To: Tom Tromey; +Cc: gdb-patches
On 11/02/2013 04:46 AM, Tom Tromey wrote:
> Yao> + # Compile source files of test case. BODY is the tcl code to do
> Yao> + # actual compilation and it should invoke 'PerfTest::compiled' if
> Yao> + # compilation is successful.
>
> Why not just have the body return success or failure and avoid having to
> call another method?
>
Yeah, we can do that. Method PerfTest::compiled and variable
compiled_ok can be removed.
> Yao> + # The top-level interface to PerfTest.
> Yao> + # COMPILE is the tcl code to generate and compile source files.
> Yao> + # STARTUP is the tcl code to start up GDB.
> Yao> + # RUN is the tcl code to drive GDB to do some operations.
> Yao> + proc assemble {compile startup run} {
>
> This approach ends up using a lot of indentation, it seems to me.
> Slimmer approaches seem possible without losing anything.
>
> E.g., in patch #4 I see:
>
> +PerfTest::assemble {
> + compile {
> + for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
> +
>
>
> But it seems to me that the "compile" part there is strictly redundant.
> We already know that this is the "compile" branch, so why not have
> "assemble" supply that?
>
> Then the above would read:
>
> PerfTest::assemble {
> for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
> ...
OK, that will simplify perftest.exp and test cases.
--
gdb/testsuite/
* lib/perftest.exp: New.
* gdb.perf/lib/perftest/__init__.py: New.
* gdb.perf/lib/perftest/measure.py: New.
* gdb.perf/lib/perftest/perftest.py: New.
* gdb.perf/lib/perftest/reporter.py: New.
* gdb.perf/lib/perftest/testresult.py: New.
---
gdb/testsuite/gdb.perf/lib/perftest/__init__.py | 17 +++
gdb/testsuite/gdb.perf/lib/perftest/measure.py | 146 +++++++++++++++++++++
gdb/testsuite/gdb.perf/lib/perftest/perftest.py | 73 ++++++++++
gdb/testsuite/gdb.perf/lib/perftest/reporter.py | 64 +++++++++
gdb/testsuite/gdb.perf/lib/perftest/testresult.py | 57 ++++++++
gdb/testsuite/lib/perftest.exp | 125 ++++++++++++++++++
6 files changed, 482 insertions(+), 0 deletions(-)
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
create mode 100644 gdb/testsuite/lib/perftest.exp
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/__init__.py b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
new file mode 100644
index 0000000..7739a3e
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
@@ -0,0 +1,17 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+"""
+GDB performance testing framework.
+"""
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
new file mode 100644
index 0000000..b7cf4e7
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import gc
+
+class Measure(object):
+ """A class that measure and collect the interesting data for a given testcase.
+
+ An instance of Measure has a collection of measurements, and each
+ of them is to measure a given aspect, such as time and memory.
+ """
+
+ def __init__(self, measurements):
+ """Constructor of measure.
+
+ measurements is a collection of Measurement objects.
+ """
+
+ self.measurements = measurements
+
+ def measure(self, func, id):
+ """Measure the operations done by func with a collection of measurements."""
+ # Enable GC, force GC and disable GC before running test in order to reduce
+ # the interference from GC.
+ gc.enable()
+ gc.collect()
+ gc.disable()
+
+ for m in self.measurements:
+ m.start(id)
+
+ func()
+
+ for m in self.measurements:
+ m.stop(id)
+
+ gc.enable()
+
+ def report(self, reporter, name):
+ """Report the measured results."""
+ for m in self.measurements:
+ m.report(reporter, name)
+
+class Measurement(object):
+ """A measurement for a certain aspect."""
+
+ def __init__(self, name, result):
+ """Constructor of Measurement.
+
+ Attribute result is the TestResult associated with measurement.
+ """
+ self.name = name;
+ self.result = result
+
+ def start(self, id):
+ """Abstract method to start the measurement."""
+ raise NotImplementedError("Abstract Method:start")
+
+ def stop(self, id):
+ """Abstract method to stop the measurement.
+
+ When the measurement is stopped, we've got something, and
+ record them in result.
+ """
+ raise NotImplementedError("Abstract Method:stop.")
+
+ def report(self, reporter, name):
+ """Report the measured data by argument reporter."""
+ self.result.report(reporter, name + " " + self.name)
+
+class MeasurementCpuTime(Measurement):
+ """Measurement on CPU time."""
+ # On UNIX, time.clock() measures the amount of CPU time that has
+ # been used by the current process. On Windows it will measure
+ # wall-clock seconds elapsed since the first call to the function.
+ # Something other than time.clock() should be used to measure CPU
+ # time on Windows.
+
+ def __init__(self, result):
+ super(MeasurementCpuTime, self).__init__("cpu_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.clock()
+
+ def stop(self, id):
+ if os.name == 'nt':
+ cpu_time = 0
+ else:
+ cpu_time = time.clock() - self.start_time
+ self.result.record (id, cpu_time)
+
+class MeasurementWallTime(Measurement):
+ """Measurement on Wall time."""
+
+ def __init__(self, result):
+ super(MeasurementWallTime, self).__init__("wall_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.time()
+
+ def stop(self, id):
+ wall_time = time.time() - self.start_time
+ self.result.record (id, wall_time)
+
+class MeasurementVmSize(Measurement):
+ """Measurement on memory usage represented by VmSize."""
+
+ def __init__(self, result):
+ super(MeasurementVmSize, self).__init__("vmsize", result)
+
+ def _compute_process_memory_usage(self, key):
+ file_path = "/proc/%d/status" % os.getpid()
+ try:
+ t = open(file_path)
+ v = t.read()
+ t.close()
+ except:
+ return 0
+ i = v.index(key)
+ v = v[i:].split(None, 3)
+ if len(v) < 3:
+ return 0
+ return int(v[1])
+
+ def start(self, id):
+ pass
+
+ def stop(self, id):
+ memory_used = self._compute_process_memory_usage("VmSize:")
+ self.result.record (id, memory_used)
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/perftest.py b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
new file mode 100644
index 0000000..e398230
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
@@ -0,0 +1,73 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import testresult
+import reporter
+from measure import Measure
+from measure import MeasurementCpuTime
+from measure import MeasurementWallTime
+from measure import MeasurementVmSize
+
+class TestCase(object):
+ """Base class of all performance testing cases.
+
+ Each sub-class should override methods execute_test, in which
+ several GDB operations are performed and measured by attribute
+ measure. Sub-class can also override method warm_up optionally
+ if the test case needs warm up.
+ """
+
+ def __init__(self, name, measure):
+ """Constructor of TestCase.
+
+ Construct an instance of TestCase with a name and a measure
+ which is to measure the test by several different measurements.
+ """
+
+ self.name = name
+ self.measure = measure
+
+ def execute_test(self):
+ """Abstract method to do the actual tests."""
+ raise NotImplementedError("Abstract Method.")
+
+ def warm_up(self):
+ """Do some operations to warm up the environment."""
+ pass
+
+ def run(self, warm_up=True, append=True):
+ """Run this test case.
+
+ It is a template method to invoke method warm_up,
+ execute_test, and finally report the measured results.
+ If parameter warm_up is True, run method warm_up. If parameter
+ append is True, the test result will be appended instead of
+ overwritten.
+ """
+ if warm_up:
+ self.warm_up()
+
+ self.execute_test()
+ self.measure.report(reporter.TextReporter(append), self.name)
+
+class TestCaseWithBasicMeasurements(TestCase):
+ """Test case measuring CPU time, wall time and memory usage."""
+
+ def __init__(self, name):
+ result_factory = testresult.SingleStatisticResultFactory()
+ measurements = [MeasurementCpuTime(result_factory.create_result()),
+ MeasurementWallTime(result_factory.create_result()),
+ MeasurementVmSize(result_factory.create_result())]
+ super (TestCaseWithBasicMeasurements, self).__init__ (name, Measure(measurements))
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
new file mode 100644
index 0000000..a902e4b
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
@@ -0,0 +1,64 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class Reporter(object):
+ """Base class of reporter to report test results in a certain format.
+
+ Subclass, which is specific to a report format, should overwrite
+ methods report, start and end.
+ """
+
+ def __init__(self, append):
+ """Constructor of Reporter.
+
+ attribute append is used to determine whether to append or
+ overwrite log file.
+ """
+ self.append = append
+
+ def report(self, *args):
+ raise NotImplementedError("Abstract Method:report.")
+
+ def start(self):
+ """Invoked when reporting is started."""
+ raise NotImplementedError("Abstract Method:start.")
+
+ def end(self):
+ """Invoked when reporting is done.
+
+ It must be overridden to do some cleanups, such as closing file
+ descriptors.
+ """
+ raise NotImplementedError("Abstract Method:end.")
+
+class TextReporter(Reporter):
+ """Report results in a plain text file 'perftest.log'."""
+
+ def __init__(self, append):
+ super (TextReporter, self).__init__(Reporter(append))
+ self.txt_log = None
+
+ def report(self, *args):
+ self.txt_log.write(' '.join(str(arg) for arg in args))
+ self.txt_log.write('\n')
+
+ def start(self):
+ if self.append:
+ self.txt_log = open ("perftest.log", 'a+');
+ else:
+ self.txt_log = open ("perftest.log", 'w');
+
+ def end(self):
+ self.txt_log.close ()
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
new file mode 100644
index 0000000..e571f12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
@@ -0,0 +1,57 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class TestResult(object):
+ """Base class to record and report test results.
+
+ Method record is to record the results of test case, and report
+ method is to report the recorded results by a given reporter.
+ """
+
+ def record(self, parameter, result):
+ raise NotImplementedError("Abstract Method:record.")
+
+ def report(self, reporter, name):
+ """Report the test results by reporter."""
+ raise NotImplementedError("Abstract Method:report.")
+
+class SingleStatisticTestResult(TestResult):
+ """Test results for the test case with a single statistic."""
+
+ def __init__(self):
+ super (SingleStatisticTestResult, self).__init__ ()
+ self.results = dict ()
+
+ def record(self, parameter, result):
+ self.results[parameter] = result
+
+ def report(self, reporter, name):
+ reporter.start()
+ for key in sorted(self.results.iterkeys()):
+ reporter.report(name, key, self.results[key])
+ reporter.end()
+
+class ResultFactory(object):
+ """A factory to create an instance of TestResult."""
+
+ def create_result(self):
+ """Create an instance of TestResult."""
+ raise NotImplementedError("Abstract Method:create_result.")
+
+class SingleStatisticResultFactory(ResultFactory):
+ """A factory to create an instance of SingleStatisticTestResult."""
+
+ def create_result(self):
+ return SingleStatisticTestResult()
diff --git a/gdb/testsuite/lib/perftest.exp b/gdb/testsuite/lib/perftest.exp
new file mode 100644
index 0000000..988e4c2
--- /dev/null
+++ b/gdb/testsuite/lib/perftest.exp
@@ -0,0 +1,125 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+namespace eval PerfTest {
+ # The name of python file on build.
+ variable remote_python_file
+
+ # A private method to set up GDB for performance testing.
+ proc _setup_perftest {} {
+ variable remote_python_file
+ global srcdir subdir testfile
+
+ set remote_python_file [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py]
+
+ # Set sys.path for module perftest.
+ gdb_test_no_output "python import os, sys"
+ gdb_test_no_output "python sys.path.insert\(0, os.path.abspath\(\"${srcdir}/${subdir}/lib\"\)\)"
+ gdb_test_no_output "python exec (open ('${remote_python_file}').read ())"
+ }
+
+ # A private method to do some cleanups when performance test is
+ # finished.
+ proc _teardown_perftest {} {
+ variable remote_python_file
+
+ remote_file host delete $remote_python_file
+ }
+
+ # Compile source files of test case. BODY is the tcl code to do
+ # actual compilation. Return zero if compilation is successful,
+ # otherwise return non-zero.
+ proc compile {body} {
+ global GDB_PERFTEST_MODE
+
+ if { [info exists GDB_PERFTEST_MODE]
+ && [string compare $GDB_PERFTEST_MODE "run"] } {
+ return [uplevel 2 $body]
+ }
+
+ return 0
+ }
+
+ # Start up GDB.
+ proc startup_gdb {body} {
+ uplevel 2 $body
+ }
+
+ # Run the performance test.
+ proc run {body} {
+ global timeout
+ global GDB_PERFTEST_TIMEOUT
+
+ set oldtimeout $timeout
+ if { [info exists GDB_PERFTEST_TIMEOUT] } {
+ set timeout $GDB_PERFTEST_TIMEOUT
+ } else {
+ set timeout 3000
+ }
+ uplevel 2 $body
+
+ set timeout $oldtimeout
+ }
+
+ # The top-level interface to PerfTest.
+ # COMPILE is the tcl code to generate and compile source files.
+ # Return zero if compilation is successful, otherwise return
+ # non-zero.
+ # STARTUP is the tcl code to start up GDB.
+ # RUN is the tcl code to drive GDB to do some operations.
+ proc assemble {compile startup run} {
+ global GDB_PERFTEST_MODE
+
+ if { [eval compile {$compile}] } {
+ untested "Could not compile source files."
+ return
+ }
+
+ # Don't execute the run if GDB_PERFTEST_MODE=compile.
+ if { [info exists GDB_PERFTEST_MODE]
+ && [string compare $GDB_PERFTEST_MODE "compile"] == 0} {
+ return
+ }
+
+ eval $startup
+
+ _setup_perftest
+
+ eval run {$run}
+
+ _teardown_perftest
+ }
+}
+
+# Return true if performance tests are skipped.
+
+proc skip_perf_tests { } {
+ global GDB_PERFTEST_MODE
+
+ if [info exists GDB_PERFTEST_MODE] {
+
+ if { "$GDB_PERFTEST_MODE" != "compile"
+ && "$GDB_PERFTEST_MODE" != "run"
+ && "$GDB_PERFTEST_MODE" != "both" } {
+ # GDB_PERFTEST_MODE=compile|run|both is allowed.
+ unsupported "Unknown value of GDB_PERFTEST_MODE."
+ return 1
+ }
+
+ return 0
+ }
+
+ return 1
+}
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-11-01 7:36 ` [PATCH 2/4] Perf test framework Yao Qi
2013-11-01 19:43 ` Doug Evans
@ 2013-11-01 20:46 ` Tom Tromey
2013-11-02 2:42 ` Yao Qi
1 sibling, 1 reply; 24+ messages in thread
From: Tom Tromey @ 2013-11-01 20:46 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
>>>>> "Yao" == Yao Qi <yao@codesourcery.com> writes:
Yao> + # Compile source files of test case. BODY is the tcl code to do
Yao> + # actual compilation and it should invoke 'PerfTest::compiled' if
Yao> + # compilation is successful.
Why not just have the body return success or failure and avoid having to
call another method?
Yao> + # The top-level interface to PerfTest.
Yao> + # COMPILE is the tcl code to generate and compile source files.
Yao> + # STARTUP is the tcl code to start up GDB.
Yao> + # RUN is the tcl code to drive GDB to do some operations.
Yao> + proc assemble {compile startup run} {
This approach ends up using a lot of indentation, it seems to me.
Slimmer approaches seem possible without losing anything.
E.g., in patch #4 I see:
+PerfTest::assemble {
+ compile {
+ for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
+
But it seems to me that the "compile" part there is strictly redundant.
We already know that this is the "compile" branch, so why not have
"assemble" supply that?
Then the above would read:
PerfTest::assemble {
for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
...
Tom
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 2/4] Perf test framework
2013-11-01 7:36 ` [PATCH 2/4] Perf test framework Yao Qi
@ 2013-11-01 19:43 ` Doug Evans
2013-11-01 20:46 ` Tom Tromey
1 sibling, 0 replies; 24+ messages in thread
From: Doug Evans @ 2013-11-01 19:43 UTC (permalink / raw)
To: Yao Qi; +Cc: gdb-patches
On Fri, Nov 1, 2013 at 12:34 AM, Yao Qi <yao@codesourcery.com> wrote:
> This patch adds a basic framework to do performance testing for GDB.
> perftest.py is about the test case, testresult.py is about test
> results, and how are they saved. reporter.py is about how results
> are reported (in what format). measure.py is about measuring the
> execution of tests by a collection of measurements.
>
> In V4:
>
> - Rename MeasurementCPUTime to MeasurementCpuTime,
> - Add 'pass' in empty method,
> - Simplify string comparison in perftest.exp.
> - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename
> GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT.
>
> In V3, there are some changes,
>
> - Add wall time measurement, cpu time measurement and vmsize
> measurement.
> - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements,
> which measures cpu time, wall time, and memory (vmsize).
> - GDB_PERFORMANCE=run|compile|both to control the mode of perf
> testing.
> - New GDB_PERFORMANCE_TIMEOUT to specify the timeout.
> - Split proc prepare to proc compile and startup.
> - Disable GC while doing measurements.
>
> In V2, there are several changes to address Doug and Sanimir's
> comments.
>
> - Add copyright header and docstring in perftest/__init__.py
> - Remove config.py.
> - Fix docstring format.
> - Rename classes "SingleVariable" to "SingleStatistic".
> - Don't extend gdb.Function in class TestCase. Add a new method run
> to run the test case so that we can pass parameters to test.
> - Allow to customize whether to warm up and to append test log.
> - Move time measurement into test harness. Add a new class
> Measurement for a specific measurement and a new class Measure to
> measure them for a given test case.
> - A new class ResultFactory to create instances of TestResult.
> - New file lib/perftest.exp, which is to do some preparations and
> cleanups to simplify each *.exp file.
> - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set.
>
> gdb/testsuite/
>
> * lib/perftest.exp: New.
> * gdb.perf/lib/perftest/__init__.py: New.
> * gdb.perf/lib/perftest/measure.py: New.
> * gdb.perf/lib/perftest/perftest.py: New.
> * gdb.perf/lib/perftest/reporter.py: New.
> * gdb.perf/lib/perftest/testresult.py: New.
Ok by me.
[Might want to give it a day or two to see if anyone else has any
final comments.]
Thanks for the perseverance!
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 2/4] Perf test framework
2013-11-01 7:36 [PATCH 0/4 V4] " Yao Qi
@ 2013-11-01 7:36 ` Yao Qi
2013-11-01 19:43 ` Doug Evans
2013-11-01 20:46 ` Tom Tromey
0 siblings, 2 replies; 24+ messages in thread
From: Yao Qi @ 2013-11-01 7:36 UTC (permalink / raw)
To: gdb-patches
This patch adds a basic framework to do performance testing for GDB.
perftest.py is about the test case, testresult.py is about test
results, and how are they saved. reporter.py is about how results
are reported (in what format). measure.py is about measuring the
execution of tests by a collection of measurements.
In V4:
- Rename MeasurementCPUTime to MeasurementCpuTime,
- Add 'pass' in empty method,
- Simplify string comparison in perftest.exp.
- Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename
GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT.
In V3, there are some changes,
- Add wall time measurement, cpu time measurement and vmsize
measurement.
- Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements,
which measures cpu time, wall time, and memory (vmsize).
- GDB_PERFORMANCE=run|compile|both to control the mode of perf
testing.
- New GDB_PERFORMANCE_TIMEOUT to specify the timeout.
- Split proc prepare to proc compile and startup.
- Disable GC while doing measurements.
In V2, there are several changes to address Doug and Sanimir's
comments.
- Add copyright header and docstring in perftest/__init__.py
- Remove config.py.
- Fix docstring format.
- Rename classes "SingleVariable" to "SingleStatistic".
- Don't extend gdb.Function in class TestCase. Add a new method run
to run the test case so that we can pass parameters to test.
- Allow to customize whether to warm up and to append test log.
- Move time measurement into test harness. Add a new class
Measurement for a specific measurement and a new class Measure to
measure them for a given test case.
- A new class ResultFactory to create instances of TestResult.
- New file lib/perftest.exp, which is to do some preparations and
cleanups to simplify each *.exp file.
- Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set.
gdb/testsuite/
* lib/perftest.exp: New.
* gdb.perf/lib/perftest/__init__.py: New.
* gdb.perf/lib/perftest/measure.py: New.
* gdb.perf/lib/perftest/perftest.py: New.
* gdb.perf/lib/perftest/reporter.py: New.
* gdb.perf/lib/perftest/testresult.py: New.
---
gdb/testsuite/gdb.perf/lib/perftest/__init__.py | 17 +++
gdb/testsuite/gdb.perf/lib/perftest/measure.py | 146 ++++++++++++++++++++
gdb/testsuite/gdb.perf/lib/perftest/perftest.py | 73 ++++++++++
gdb/testsuite/gdb.perf/lib/perftest/reporter.py | 64 +++++++++
gdb/testsuite/gdb.perf/lib/perftest/testresult.py | 57 ++++++++
gdb/testsuite/lib/perftest.exp | 148 +++++++++++++++++++++
6 files changed, 505 insertions(+), 0 deletions(-)
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
create mode 100644 gdb/testsuite/lib/perftest.exp
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/__init__.py b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
new file mode 100644
index 0000000..7739a3e
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
@@ -0,0 +1,17 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+"""
+GDB performance testing framework.
+"""
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
new file mode 100644
index 0000000..b7cf4e7
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import gc
+
+class Measure(object):
+ """A class that measure and collect the interesting data for a given testcase.
+
+ An instance of Measure has a collection of measurements, and each
+ of them is to measure a given aspect, such as time and memory.
+ """
+
+ def __init__(self, measurements):
+ """Constructor of measure.
+
+ measurements is a collection of Measurement objects.
+ """
+
+ self.measurements = measurements
+
+ def measure(self, func, id):
+ """Measure the operations done by func with a collection of measurements."""
+ # Enable GC, force GC and disable GC before running test in order to reduce
+ # the interference from GC.
+ gc.enable()
+ gc.collect()
+ gc.disable()
+
+ for m in self.measurements:
+ m.start(id)
+
+ func()
+
+ for m in self.measurements:
+ m.stop(id)
+
+ gc.enable()
+
+ def report(self, reporter, name):
+ """Report the measured results."""
+ for m in self.measurements:
+ m.report(reporter, name)
+
+class Measurement(object):
+ """A measurement for a certain aspect."""
+
+ def __init__(self, name, result):
+ """Constructor of Measurement.
+
+ Attribute result is the TestResult associated with measurement.
+ """
+ self.name = name;
+ self.result = result
+
+ def start(self, id):
+ """Abstract method to start the measurement."""
+ raise NotImplementedError("Abstract Method:start")
+
+ def stop(self, id):
+ """Abstract method to stop the measurement.
+
+ When the measurement is stopped, we've got something, and
+ record them in result.
+ """
+ raise NotImplementedError("Abstract Method:stop.")
+
+ def report(self, reporter, name):
+ """Report the measured data by argument reporter."""
+ self.result.report(reporter, name + " " + self.name)
+
+class MeasurementCpuTime(Measurement):
+ """Measurement on CPU time."""
+ # On UNIX, time.clock() measures the amount of CPU time that has
+ # been used by the current process. On Windows it will measure
+ # wall-clock seconds elapsed since the first call to the function.
+ # Something other than time.clock() should be used to measure CPU
+ # time on Windows.
+
+ def __init__(self, result):
+ super(MeasurementCpuTime, self).__init__("cpu_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.clock()
+
+ def stop(self, id):
+ if os.name == 'nt':
+ cpu_time = 0
+ else:
+ cpu_time = time.clock() - self.start_time
+ self.result.record (id, cpu_time)
+
+class MeasurementWallTime(Measurement):
+ """Measurement on Wall time."""
+
+ def __init__(self, result):
+ super(MeasurementWallTime, self).__init__("wall_time", result)
+ self.start_time = 0
+
+ def start(self, id):
+ self.start_time = time.time()
+
+ def stop(self, id):
+ wall_time = time.time() - self.start_time
+ self.result.record (id, wall_time)
+
+class MeasurementVmSize(Measurement):
+ """Measurement on memory usage represented by VmSize."""
+
+ def __init__(self, result):
+ super(MeasurementVmSize, self).__init__("vmsize", result)
+
+ def _compute_process_memory_usage(self, key):
+ file_path = "/proc/%d/status" % os.getpid()
+ try:
+ t = open(file_path)
+ v = t.read()
+ t.close()
+ except:
+ return 0
+ i = v.index(key)
+ v = v[i:].split(None, 3)
+ if len(v) < 3:
+ return 0
+ return int(v[1])
+
+ def start(self, id):
+ pass
+
+ def stop(self, id):
+ memory_used = self._compute_process_memory_usage("VmSize:")
+ self.result.record (id, memory_used)
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/perftest.py b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
new file mode 100644
index 0000000..e398230
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
@@ -0,0 +1,73 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import testresult
+import reporter
+from measure import Measure
+from measure import MeasurementCpuTime
+from measure import MeasurementWallTime
+from measure import MeasurementVmSize
+
+class TestCase(object):
+ """Base class of all performance testing cases.
+
+ Each sub-class should override methods execute_test, in which
+ several GDB operations are performed and measured by attribute
+ measure. Sub-class can also override method warm_up optionally
+ if the test case needs warm up.
+ """
+
+ def __init__(self, name, measure):
+ """Constructor of TestCase.
+
+ Construct an instance of TestCase with a name and a measure
+ which is to measure the test by several different measurements.
+ """
+
+ self.name = name
+ self.measure = measure
+
+ def execute_test(self):
+ """Abstract method to do the actual tests."""
+ raise NotImplementedError("Abstract Method.")
+
+ def warm_up(self):
+ """Do some operations to warm up the environment."""
+ pass
+
+ def run(self, warm_up=True, append=True):
+ """Run this test case.
+
+ It is a template method to invoke method warm_up,
+ execute_test, and finally report the measured results.
+ If parameter warm_up is True, run method warm_up. If parameter
+ append is True, the test result will be appended instead of
+ overwritten.
+ """
+ if warm_up:
+ self.warm_up()
+
+ self.execute_test()
+ self.measure.report(reporter.TextReporter(append), self.name)
+
+class TestCaseWithBasicMeasurements(TestCase):
+ """Test case measuring CPU time, wall time and memory usage."""
+
+ def __init__(self, name):
+ result_factory = testresult.SingleStatisticResultFactory()
+ measurements = [MeasurementCpuTime(result_factory.create_result()),
+ MeasurementWallTime(result_factory.create_result()),
+ MeasurementVmSize(result_factory.create_result())]
+ super (TestCaseWithBasicMeasurements, self).__init__ (name, Measure(measurements))
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
new file mode 100644
index 0000000..a902e4b
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
@@ -0,0 +1,64 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class Reporter(object):
+ """Base class of reporter to report test results in a certain format.
+
+ Subclass, which is specific to a report format, should overwrite
+ methods report, start and end.
+ """
+
+ def __init__(self, append):
+ """Constructor of Reporter.
+
+ attribute append is used to determine whether to append or
+ overwrite log file.
+ """
+ self.append = append
+
+ def report(self, *args):
+ raise NotImplementedError("Abstract Method:report.")
+
+ def start(self):
+ """Invoked when reporting is started."""
+ raise NotImplementedError("Abstract Method:start.")
+
+ def end(self):
+ """Invoked when reporting is done.
+
+ It must be overridden to do some cleanups, such as closing file
+ descriptors.
+ """
+ raise NotImplementedError("Abstract Method:end.")
+
+class TextReporter(Reporter):
+ """Report results in a plain text file 'perftest.log'."""
+
+ def __init__(self, append):
+ super (TextReporter, self).__init__(Reporter(append))
+ self.txt_log = None
+
+ def report(self, *args):
+ self.txt_log.write(' '.join(str(arg) for arg in args))
+ self.txt_log.write('\n')
+
+ def start(self):
+ if self.append:
+ self.txt_log = open ("perftest.log", 'a+');
+ else:
+ self.txt_log = open ("perftest.log", 'w');
+
+ def end(self):
+ self.txt_log.close ()
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
new file mode 100644
index 0000000..e571f12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
@@ -0,0 +1,57 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+class TestResult(object):
+ """Base class to record and report test results.
+
+ Method record is to record the results of test case, and report
+ method is to report the recorded results by a given reporter.
+ """
+
+ def record(self, parameter, result):
+ raise NotImplementedError("Abstract Method:record.")
+
+ def report(self, reporter, name):
+ """Report the test results by reporter."""
+ raise NotImplementedError("Abstract Method:report.")
+
+class SingleStatisticTestResult(TestResult):
+ """Test results for the test case with a single statistic."""
+
+ def __init__(self):
+ super (SingleStatisticTestResult, self).__init__ ()
+ self.results = dict ()
+
+ def record(self, parameter, result):
+ self.results[parameter] = result
+
+ def report(self, reporter, name):
+ reporter.start()
+ for key in sorted(self.results.iterkeys()):
+ reporter.report(name, key, self.results[key])
+ reporter.end()
+
+class ResultFactory(object):
+ """A factory to create an instance of TestResult."""
+
+ def create_result(self):
+ """Create an instance of TestResult."""
+ raise NotImplementedError("Abstract Method:create_result.")
+
+class SingleStatisticResultFactory(ResultFactory):
+ """A factory to create an instance of SingleStatisticTestResult."""
+
+ def create_result(self):
+ return SingleStatisticTestResult()
diff --git a/gdb/testsuite/lib/perftest.exp b/gdb/testsuite/lib/perftest.exp
new file mode 100644
index 0000000..0c489e8
--- /dev/null
+++ b/gdb/testsuite/lib/perftest.exp
@@ -0,0 +1,148 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+namespace eval PerfTest {
+ # The name of python file on build.
+ variable remote_python_file
+
+ # The source files are compiled successfully or not.
+ variable compiled_ok
+
+ # A private method to set up GDB for performance testing.
+ proc _setup_perftest {} {
+ variable remote_python_file
+ global srcdir subdir testfile
+
+ set remote_python_file [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py]
+
+ # Set sys.path for module perftest.
+ gdb_test_no_output "python import os, sys"
+ gdb_test_no_output "python sys.path.insert\(0, os.path.abspath\(\"${srcdir}/${subdir}/lib\"\)\)"
+ gdb_test_no_output "python exec (open ('${remote_python_file}').read ())"
+ }
+
+ # A private method to do some cleanups when performance test is
+ # finished.
+ proc _teardown_perftest {} {
+ variable remote_python_file
+
+ remote_file host delete $remote_python_file
+ }
+
+ # Compile source files of test case. BODY is the tcl code to do
+ # actual compilation and it should invoke 'PerfTest::compiled' if
+ # compilation is successful.
+ proc compile {body} {
+ global GDB_PERFTEST_MODE
+
+ if { [info exists GDB_PERFTEST_MODE]
+ && [string compare $GDB_PERFTEST_MODE "run"] } {
+ variable compiled_ok
+
+ set compiled_ok 0
+ uplevel 2 $body
+
+ if {!$compiled_ok} {
+ untested "Could not compile source files."
+ return -1
+ }
+ }
+ }
+
+ # Mark the compilation is finished successfully.
+ proc compiled {} {
+ variable compiled_ok
+
+ set compiled_ok 1
+ }
+
+ # Start up GDB.
+ proc startup_gdb {body} {
+ variable compiled_ok
+
+ if {!$compiled_ok} {
+ return
+ }
+
+ uplevel 2 $body
+ }
+
+ # Run the performance test.
+ proc run {body} {
+ global timeout
+ global GDB_PERFTEST_TIMEOUT
+
+ set oldtimeout $timeout
+ if { [info exists GDB_PERFTEST_TIMEOUT] } {
+ set timeout $GDB_PERFTEST_TIMEOUT
+ } else {
+ set timeout 3000
+ }
+ uplevel 2 $body
+
+ set timeout $oldtimeout
+ }
+
+ # The top-level interface to PerfTest.
+ # COMPILE is the tcl code to generate and compile source files.
+ # STARTUP is the tcl code to start up GDB.
+ # RUN is the tcl code to drive GDB to do some operations.
+ proc assemble {compile startup run} {
+ variable compiled_ok
+ global GDB_PERFTEST_MODE
+
+ set compiled_ok 1
+ eval $compile
+
+ if {!$compiled_ok} {
+ return
+ }
+
+ # Don't execute the run if GDB_PERFTEST_MODE=compile.
+ if { [info exists GDB_PERFTEST_MODE]
+ && [string compare $GDB_PERFTEST_MODE "compile"] == 0} {
+ return
+ }
+
+ eval $startup
+
+ _setup_perftest
+
+ eval $run
+
+ _teardown_perftest
+ }
+}
+
+# Return true if performance tests are skipped.
+
+proc skip_perf_tests { } {
+ global GDB_PERFTEST_MODE
+
+ if [info exists GDB_PERFTEST_MODE] {
+
+ if { "$GDB_PERFTEST_MODE" != "compile"
+ && "$GDB_PERFTEST_MODE" != "run"
+ && "$GDB_PERFTEST_MODE" != "both" } {
+ # GDB_PERFTEST_MODE=compile|run|both is allowed.
+ unsupported "Unknown value of GDB_PERFTEST_MODE."
+ return 1
+ }
+
+ return 0
+ }
+
+ return 1
+}
--
1.7.7.6
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2013-11-06 5:24 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-10-16 7:10 [PATCH 0/4 V3] GDB Performance testing Yao Qi
2013-10-16 7:10 ` [PATCH 4/4] Test on solib load and unload Yao Qi
2013-10-25 7:17 ` Doug Evans
2013-10-16 7:10 ` [PATCH 3/4] Mention perf test in testsuite/README Yao Qi
2013-10-25 6:43 ` Doug Evans
2013-10-16 7:10 ` [PATCH 2/4] Perf test framework Yao Qi
2013-10-16 9:05 ` Yao Qi
2013-10-24 14:20 ` Yao Qi
2013-10-25 6:33 ` Doug Evans
2013-10-25 6:32 ` Doug Evans
2013-10-25 8:57 ` Yao Qi
2013-10-16 7:10 ` [PATCH 1/4] New make target 'check-perf' and new dir gdb.perf Yao Qi
2013-10-25 3:04 ` Doug Evans
2013-10-25 17:34 ` Pedro Alves
2013-10-24 2:44 ` ping: [PATCH 0/4 V3] GDB Performance testing Yao Qi
2013-10-24 8:50 ` Agovic, Sanimir
2013-10-24 14:13 ` Yao Qi
2013-10-25 2:59 ` Doug Evans
2013-11-01 7:36 [PATCH 0/4 V4] " Yao Qi
2013-11-01 7:36 ` [PATCH 2/4] Perf test framework Yao Qi
2013-11-01 19:43 ` Doug Evans
2013-11-01 20:46 ` Tom Tromey
2013-11-02 2:42 ` Yao Qi
2013-11-04 16:27 ` Tom Tromey
2013-11-06 5:30 ` Yao Qi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).