[PATCH 1/4] New make target 'check-perf' and new dir gdb.perf

[PATCH 1/4] New make target 'check-perf' and new dir gdb.perf

[Yao Qi]

We add a new dir gdb.perf in testsuite for all performance tests.
However, current 'make check' logic will either run dejagnu in
directory testsuite or iterate all gdb.* directories which has *.exp
files.  Both of them will run tests in gdb.perf.  We want to achieve:

 1) typical 'make check' should not run performance tests.  In each perf
    test case, GDB_PERFORMANCE is checked.  If it doesn't exist, return.
 2) run perf tests easily.  We add a new makefile target 'check-perf'.

V2 is simpler than V1, since we don't have to filter out gdb.perf
directory.

V3: Move GDB_PERFORMANCE=both appears before RUNTESTFLAGS.  It was
approved by Doug.

V4: Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE

gdb:

2013-11-01  Yao Qi  <yao@codesourcery.com>

	* Makefile.in (check-perf): New target.

gdb/testsuite:

2013-11-01  Yao Qi  <yao@codesourcery.com>

	* Makefile.in (check-perf): New target.
	* configure.ac (AC_OUTPUT): Output Makefile in gdb.perf.
	* configure: Re-generated.
	* gdb.perf/Makefile.in: New.
---
 gdb/Makefile.in                    |    8 ++++++++
 gdb/testsuite/Makefile.in          |    4 ++++
 gdb/testsuite/configure            |    3 ++-
 gdb/testsuite/configure.ac         |    2 +-
 gdb/testsuite/gdb.perf/Makefile.in |   15 +++++++++++++++
 5 files changed, 30 insertions(+), 2 deletions(-)
 create mode 100644 gdb/testsuite/gdb.perf/Makefile.in

diff --git a/gdb/Makefile.in b/gdb/Makefile.in
index a9b3c64..011b1aa 100644
--- a/gdb/Makefile.in
+++ b/gdb/Makefile.in
@@ -1009,6 +1009,14 @@ check: force
 	  $(MAKE) $(TARGET_FLAGS_TO_PASS) check; \
 	else true; fi
 
+check-perf: force
+	@if [ -f testsuite/Makefile ]; then \
+	  rootme=`pwd`; export rootme; \
+	  rootsrc=`cd $(srcdir); pwd`; export rootsrc; \
+	  cd testsuite; \
+	  $(MAKE) $(TARGET_FLAGS_TO_PASS) check-perf; \
+	else true; fi
+
 # The idea is to parallelize testing of multilibs, for example:
 #   make -j3 check//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu}
 # will run 3 concurrent sessions of check, eventually testing all 10
diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in
index a7b3d5c..e7c2b50 100644
--- a/gdb/testsuite/Makefile.in
+++ b/gdb/testsuite/Makefile.in
@@ -187,6 +187,10 @@ check-gdb.base%: all $(abs_builddir)/site.exp
 	@if test ! -d gdb.base$*; then mkdir gdb.base$*; fi
 	$(DO_RUNTEST) $(BASE$*_FILES) --outdir gdb.base$* $(RUNTESTFLAGS)
 
+check-perf: all $(abs_builddir)/site.exp
+	@if test ! -d gdb.perf; then mkdir gdb.perf; fi
+	$(DO_RUNTEST) --directory=gdb.perf --outdir gdb.perf GDB_PERFTEST_MODE=both $(RUNTESTFLAGS)
+
 subdir_do: force
 	@for i in $(DODIRS); do \
 		if [ -d ./$$i ] ; then \
diff --git a/gdb/testsuite/configure b/gdb/testsuite/configure
index a40c144..da590f3 100755
--- a/gdb/testsuite/configure
+++ b/gdb/testsuite/configure
@@ -3448,7 +3448,7 @@ done
 
 
 
-ac_config_files="$ac_config_files Makefile gdb.ada/Makefile gdb.arch/Makefile gdb.asm/Makefile gdb.base/Makefile gdb.btrace/Makefile gdb.cell/Makefile gdb.cp/Makefile gdb.disasm/Makefile gdb.dwarf2/Makefile gdb.fortran/Makefile gdb.go/Makefile gdb.server/Makefile gdb.java/Makefile gdb.hp/Makefile gdb.hp/gdb.objdbg/Makefile gdb.hp/gdb.base-hp/Makefile gdb.hp/gdb.aCC/Makefile gdb.hp/gdb.compat/Makefile gdb.hp/gdb.defects/Makefile gdb.linespec/Makefile gdb.mi/Makefile gdb.modula2/Makefile gdb.multi/Makefile gdb.objc/Makefile gdb.opencl/Makefile gdb.opt/Makefile gdb.pascal/Makefile gdb.python/Makefile gdb.reverse/Makefile gdb.stabs/Makefile gdb.threads/Makefile gdb.trace/Makefile gdb.xml/Makefile"
+ac_config_files="$ac_config_files Makefile gdb.ada/Makefile gdb.arch/Makefile gdb.asm/Makefile gdb.base/Makefile gdb.btrace/Makefile gdb.cell/Makefile gdb.cp/Makefile gdb.disasm/Makefile gdb.dwarf2/Makefile gdb.fortran/Makefile gdb.go/Makefile gdb.server/Makefile gdb.java/Makefile gdb.hp/Makefile gdb.hp/gdb.objdbg/Makefile gdb.hp/gdb.base-hp/Makefile gdb.hp/gdb.aCC/Makefile gdb.hp/gdb.compat/Makefile gdb.hp/gdb.defects/Makefile gdb.linespec/Makefile gdb.mi/Makefile gdb.modula2/Makefile gdb.multi/Makefile gdb.objc/Makefile gdb.opencl/Makefile gdb.opt/Makefile gdb.pascal/Makefile gdb.perf/Makefile gdb.python/Makefile gdb.reverse/Makefile gdb.stabs/Makefile gdb.threads/Makefile gdb.trace/Makefile gdb.xml/Makefile"
 
 cat >confcache <<\_ACEOF
 # This file is a shell script that caches the results of configure
@@ -4176,6 +4176,7 @@ do
     "gdb.opencl/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.opencl/Makefile" ;;
     "gdb.opt/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.opt/Makefile" ;;
     "gdb.pascal/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.pascal/Makefile" ;;
+    "gdb.perf/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.perf/Makefile" ;;
     "gdb.python/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.python/Makefile" ;;
     "gdb.reverse/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.reverse/Makefile" ;;
     "gdb.stabs/Makefile") CONFIG_FILES="$CONFIG_FILES gdb.stabs/Makefile" ;;
diff --git a/gdb/testsuite/configure.ac b/gdb/testsuite/configure.ac
index 9e07021..94f96cc 100644
--- a/gdb/testsuite/configure.ac
+++ b/gdb/testsuite/configure.ac
@@ -97,5 +97,5 @@ AC_OUTPUT([Makefile \
   gdb.hp/gdb.defects/Makefile gdb.linespec/Makefile \
   gdb.mi/Makefile gdb.modula2/Makefile gdb.multi/Makefile \
   gdb.objc/Makefile gdb.opencl/Makefile gdb.opt/Makefile gdb.pascal/Makefile \
-  gdb.python/Makefile gdb.reverse/Makefile gdb.stabs/Makefile \
+  gdb.perf/Makefile gdb.python/Makefile gdb.reverse/Makefile gdb.stabs/Makefile \
   gdb.threads/Makefile gdb.trace/Makefile gdb.xml/Makefile])
diff --git a/gdb/testsuite/gdb.perf/Makefile.in b/gdb/testsuite/gdb.perf/Makefile.in
new file mode 100644
index 0000000..2071d12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/Makefile.in
@@ -0,0 +1,15 @@
+VPATH = @srcdir@
+srcdir = @srcdir@
+
+.PHONY: all clean mostlyclean distclean realclean
+
+PROGS = 
+
+all info install-info dvi install uninstall installcheck check:
+	@echo "Nothing to be done for $@..."
+
+clean mostlyclean:
+	-rm -f *.o *.diff *~ core $(PROGS)
+
+distclean maintainer-clean realclean: clean
+	-rm -f Makefile config.status config.log gdb.log gdb.sum
-- 
1.7.7.6

[PATCH 2/4] Perf test framework

[Yao Qi]

This patch adds a basic framework to do performance testing for GDB.
perftest.py is about the test case, testresult.py is about test
results, and how are they saved.  reporter.py is about how results
are reported (in what format).  measure.py is about measuring the
execution of tests by a collection of measurements.

In V4:

 - Rename MeasurementCPUTime to MeasurementCpuTime,
 - Add 'pass' in empty method,
 - Simplify string comparison in perftest.exp.
 - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename
   GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT.

In V3, there are some changes,

 - Add wall time measurement, cpu time measurement and vmsize
   measurement.
 - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements,
   which measures cpu time, wall time, and memory (vmsize).
 - GDB_PERFORMANCE=run|compile|both to control the mode of perf
   testing.
 - New GDB_PERFORMANCE_TIMEOUT to specify the timeout.
 - Split proc prepare to proc compile and startup.
 - Disable GC while doing measurements.

In V2, there are several changes to address Doug and Sanimir's
comments.

 - Add copyright header and docstring in perftest/__init__.py
 - Remove config.py.
 - Fix docstring format.
 - Rename classes "SingleVariable" to "SingleStatistic".
 - Don't extend gdb.Function in class TestCase.  Add a new method run
   to run the test case so that we can pass parameters to test.
 - Allow to customize whether to warm up and to append test log.
 - Move time measurement into test harness.  Add a new class
   Measurement for a specific measurement and a new class Measure to
   measure them for a given test case.
 - A new class ResultFactory to create instances of TestResult.
 - New file lib/perftest.exp, which is to do some preparations and
   cleanups to simplify each *.exp file.
 - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set.

gdb/testsuite/

	* lib/perftest.exp: New.
	* gdb.perf/lib/perftest/__init__.py: New.
	* gdb.perf/lib/perftest/measure.py: New.
	* gdb.perf/lib/perftest/perftest.py: New.
	* gdb.perf/lib/perftest/reporter.py: New.
	* gdb.perf/lib/perftest/testresult.py: New.
---
 gdb/testsuite/gdb.perf/lib/perftest/__init__.py   |   17 +++
 gdb/testsuite/gdb.perf/lib/perftest/measure.py    |  146 ++++++++++++++++++++
 gdb/testsuite/gdb.perf/lib/perftest/perftest.py   |   73 ++++++++++
 gdb/testsuite/gdb.perf/lib/perftest/reporter.py   |   64 +++++++++
 gdb/testsuite/gdb.perf/lib/perftest/testresult.py |   57 ++++++++
 gdb/testsuite/lib/perftest.exp                    |  148 +++++++++++++++++++++
 6 files changed, 505 insertions(+), 0 deletions(-)
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
 create mode 100644 gdb/testsuite/lib/perftest.exp

diff --git a/gdb/testsuite/gdb.perf/lib/perftest/__init__.py b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
new file mode 100644
index 0000000..7739a3e
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
@@ -0,0 +1,17 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+GDB performance testing framework.
+"""
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
new file mode 100644
index 0000000..b7cf4e7
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import gc
+
+class Measure(object):
+    """A class that measure and collect the interesting data for a given testcase.
+
+    An instance of Measure has a collection of measurements, and each
+    of them is to measure a given aspect, such as time and memory.
+    """
+
+    def __init__(self, measurements):
+        """Constructor of measure.
+
+        measurements is a collection of Measurement objects.
+        """
+
+        self.measurements = measurements
+
+    def measure(self, func, id):
+        """Measure the operations done by func with a collection of measurements."""
+        # Enable GC, force GC and disable GC before running test in order to reduce
+        # the interference from GC.
+        gc.enable()
+        gc.collect()
+        gc.disable()
+
+        for m in self.measurements:
+            m.start(id)
+
+        func()
+
+        for m in self.measurements:
+            m.stop(id)
+
+        gc.enable()
+
+    def report(self, reporter, name):
+        """Report the measured results."""
+        for m in self.measurements:
+            m.report(reporter, name)
+
+class Measurement(object):
+    """A measurement for a certain aspect."""
+
+    def __init__(self, name, result):
+        """Constructor of Measurement.
+
+        Attribute result is the TestResult associated with measurement.
+        """
+        self.name = name;
+        self.result = result
+
+    def start(self, id):
+        """Abstract method to start the measurement."""
+        raise NotImplementedError("Abstract Method:start")
+
+    def stop(self, id):
+        """Abstract method to stop the measurement.
+
+        When the measurement is stopped, we've got something, and
+        record them in result.
+        """
+        raise NotImplementedError("Abstract Method:stop.")
+
+    def report(self, reporter, name):
+        """Report the measured data by argument reporter."""
+        self.result.report(reporter, name + " " + self.name)
+
+class MeasurementCpuTime(Measurement):
+    """Measurement on CPU time."""
+    # On UNIX, time.clock() measures the amount of CPU time that has
+    # been used by the current process.  On Windows it will measure
+    # wall-clock seconds elapsed since the first call to the function.
+    # Something other than time.clock() should be used to measure CPU
+    # time on Windows.
+
+    def __init__(self, result):
+        super(MeasurementCpuTime, self).__init__("cpu_time", result)
+        self.start_time = 0
+
+    def start(self, id):
+        self.start_time = time.clock()
+
+    def stop(self, id):
+        if os.name == 'nt':
+            cpu_time = 0
+        else:
+            cpu_time = time.clock() - self.start_time
+        self.result.record (id, cpu_time)
+
+class MeasurementWallTime(Measurement):
+    """Measurement on Wall time."""
+
+    def __init__(self, result):
+        super(MeasurementWallTime, self).__init__("wall_time", result)
+        self.start_time = 0
+
+    def start(self, id):
+        self.start_time = time.time()
+
+    def stop(self, id):
+        wall_time = time.time() - self.start_time
+        self.result.record (id, wall_time)
+
+class MeasurementVmSize(Measurement):
+    """Measurement on memory usage represented by VmSize."""
+
+    def __init__(self, result):
+        super(MeasurementVmSize, self).__init__("vmsize", result)
+
+    def _compute_process_memory_usage(self, key):
+        file_path = "/proc/%d/status" % os.getpid()
+        try:
+            t = open(file_path)
+            v = t.read()
+            t.close()
+        except:
+            return 0
+        i = v.index(key)
+        v = v[i:].split(None, 3)
+        if len(v) < 3:
+            return 0
+        return int(v[1])
+
+    def start(self, id):
+        pass
+
+    def stop(self, id):
+        memory_used = self._compute_process_memory_usage("VmSize:")
+        self.result.record (id, memory_used)
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/perftest.py b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
new file mode 100644
index 0000000..e398230
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
@@ -0,0 +1,73 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import testresult
+import reporter
+from measure import Measure
+from measure import MeasurementCpuTime
+from measure import MeasurementWallTime
+from measure import MeasurementVmSize
+
+class TestCase(object):
+    """Base class of all performance testing cases.
+
+    Each sub-class should override methods execute_test, in which
+    several GDB operations are performed and measured by attribute
+    measure.  Sub-class can also override method warm_up optionally
+    if the test case needs warm up.
+    """
+
+    def __init__(self, name, measure):
+        """Constructor of TestCase.
+
+        Construct an instance of TestCase with a name and a measure
+        which is to measure the test by several different measurements.
+        """
+
+        self.name = name
+        self.measure = measure
+
+    def execute_test(self):
+        """Abstract method to do the actual tests."""
+        raise NotImplementedError("Abstract Method.")
+
+    def warm_up(self):
+        """Do some operations to warm up the environment."""
+        pass
+
+    def run(self, warm_up=True, append=True):
+        """Run this test case.
+
+        It is a template method to invoke method warm_up,
+        execute_test, and finally report the measured results.
+        If parameter warm_up is True, run method warm_up.  If parameter
+        append is True, the test result will be appended instead of
+        overwritten.
+        """
+        if warm_up:
+            self.warm_up()
+
+        self.execute_test()
+        self.measure.report(reporter.TextReporter(append), self.name)
+
+class TestCaseWithBasicMeasurements(TestCase):
+    """Test case measuring CPU time, wall time and memory usage."""
+
+    def __init__(self, name):
+        result_factory = testresult.SingleStatisticResultFactory()
+        measurements = [MeasurementCpuTime(result_factory.create_result()),
+                        MeasurementWallTime(result_factory.create_result()),
+                        MeasurementVmSize(result_factory.create_result())]
+        super (TestCaseWithBasicMeasurements, self).__init__ (name, Measure(measurements))
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
new file mode 100644
index 0000000..a902e4b
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
@@ -0,0 +1,64 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+class Reporter(object):
+    """Base class of reporter to report test results in a certain format.
+
+    Subclass, which is specific to a report format, should overwrite
+    methods report, start and end.
+    """
+
+    def __init__(self, append):
+        """Constructor of Reporter.
+
+        attribute append is used to determine whether to append or
+        overwrite log file.
+        """
+        self.append = append
+
+    def report(self, *args):
+        raise NotImplementedError("Abstract Method:report.")
+
+    def start(self):
+        """Invoked when reporting is started."""
+        raise NotImplementedError("Abstract Method:start.")
+
+    def end(self):
+        """Invoked when reporting is done.
+
+        It must be overridden to do some cleanups, such as closing file
+        descriptors.
+        """
+        raise NotImplementedError("Abstract Method:end.")
+
+class TextReporter(Reporter):
+    """Report results in a plain text file 'perftest.log'."""
+
+    def __init__(self, append):
+        super (TextReporter, self).__init__(Reporter(append))
+        self.txt_log = None
+
+    def report(self, *args):
+        self.txt_log.write(' '.join(str(arg) for arg in args))
+        self.txt_log.write('\n')
+
+    def start(self):
+        if self.append:
+            self.txt_log = open ("perftest.log", 'a+');
+        else:
+            self.txt_log = open ("perftest.log", 'w');
+
+    def end(self):
+        self.txt_log.close ()
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
new file mode 100644
index 0000000..e571f12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
@@ -0,0 +1,57 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+class TestResult(object):
+    """Base class to record and report test results.
+
+    Method record is to record the results of test case, and report
+    method is to report the recorded results by a given reporter.
+    """
+
+    def record(self, parameter, result):
+        raise NotImplementedError("Abstract Method:record.")
+
+    def report(self, reporter, name):
+        """Report the test results by reporter."""
+        raise NotImplementedError("Abstract Method:report.")
+
+class SingleStatisticTestResult(TestResult):
+    """Test results for the test case with a single statistic."""
+
+    def __init__(self):
+        super (SingleStatisticTestResult, self).__init__ ()
+        self.results = dict ()
+
+    def record(self, parameter, result):
+        self.results[parameter] = result
+
+    def report(self, reporter, name):
+        reporter.start()
+        for key in sorted(self.results.iterkeys()):
+            reporter.report(name, key, self.results[key])
+        reporter.end()
+
+class ResultFactory(object):
+    """A factory to create an instance of TestResult."""
+
+    def create_result(self):
+        """Create an instance of TestResult."""
+        raise NotImplementedError("Abstract Method:create_result.")
+
+class SingleStatisticResultFactory(ResultFactory):
+    """A factory to create an instance of SingleStatisticTestResult."""
+
+    def create_result(self):
+        return SingleStatisticTestResult()
diff --git a/gdb/testsuite/lib/perftest.exp b/gdb/testsuite/lib/perftest.exp
new file mode 100644
index 0000000..0c489e8
--- /dev/null
+++ b/gdb/testsuite/lib/perftest.exp
@@ -0,0 +1,148 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace eval PerfTest {
+    # The name of python file on build.
+    variable remote_python_file
+
+    # The source files are compiled successfully or not.
+    variable compiled_ok
+
+    # A private method to set up GDB for performance testing.
+    proc _setup_perftest {} {
+	variable remote_python_file
+	global srcdir subdir testfile
+
+	set remote_python_file [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py]
+
+	# Set sys.path for module perftest.
+	gdb_test_no_output "python import os, sys"
+	gdb_test_no_output "python sys.path.insert\(0, os.path.abspath\(\"${srcdir}/${subdir}/lib\"\)\)"
+	gdb_test_no_output "python exec (open ('${remote_python_file}').read ())"
+    }
+
+    # A private method to do some cleanups when performance test is
+    # finished.
+    proc _teardown_perftest {} {
+	variable remote_python_file
+
+	remote_file host delete $remote_python_file
+    }
+
+    # Compile source files of test case.  BODY is the tcl code to do
+    # actual compilation and it should invoke 'PerfTest::compiled' if
+    # compilation is successful.
+    proc compile {body} {
+	global GDB_PERFTEST_MODE
+
+	if { [info exists GDB_PERFTEST_MODE]
+	     && [string compare $GDB_PERFTEST_MODE "run"] } {
+	    variable compiled_ok
+
+	    set compiled_ok 0
+	    uplevel 2 $body
+
+	    if {!$compiled_ok} {
+		untested "Could not compile source files."
+		return -1
+	    }
+	}
+    }
+
+    # Mark the compilation is finished successfully.
+    proc compiled {} {
+	variable compiled_ok
+
+	set compiled_ok 1
+    }
+
+    # Start up GDB.
+    proc startup_gdb {body} {
+	variable compiled_ok
+
+	if {!$compiled_ok} {
+	    return
+	}
+
+	uplevel 2 $body
+    }
+
+    # Run the performance test.
+    proc run {body} {
+	global timeout
+	global GDB_PERFTEST_TIMEOUT
+
+	set oldtimeout $timeout
+	if { [info exists GDB_PERFTEST_TIMEOUT] } {
+	    set timeout $GDB_PERFTEST_TIMEOUT
+	} else {
+	    set timeout 3000
+	}
+	uplevel 2 $body
+
+	set timeout $oldtimeout
+    }
+
+    # The top-level interface to PerfTest.
+    # COMPILE is the tcl code to generate and compile source files.
+    # STARTUP is the tcl code to start up GDB.
+    # RUN is the tcl code to drive GDB to do some operations.
+    proc assemble {compile startup run} {
+	variable compiled_ok
+	global GDB_PERFTEST_MODE
+
+	set compiled_ok 1
+	eval $compile
+
+	if {!$compiled_ok} {
+	    return
+	}
+
+	# Don't execute the run if GDB_PERFTEST_MODE=compile.
+	if { [info exists GDB_PERFTEST_MODE]
+	     && [string compare $GDB_PERFTEST_MODE "compile"] == 0} {
+	    return
+	}
+
+	eval $startup
+
+	_setup_perftest
+
+	eval $run
+
+	_teardown_perftest
+    }
+}
+
+# Return true if performance tests are skipped.
+
+proc skip_perf_tests { } {
+    global GDB_PERFTEST_MODE
+
+    if [info exists GDB_PERFTEST_MODE] {
+
+	if { "$GDB_PERFTEST_MODE" != "compile"
+	     && "$GDB_PERFTEST_MODE" != "run"
+	     && "$GDB_PERFTEST_MODE" != "both" } {
+	    # GDB_PERFTEST_MODE=compile|run|both is allowed.
+	    unsupported "Unknown value of GDB_PERFTEST_MODE."
+	    return 1
+	}
+
+	return 0
+    }
+
+    return 1
+}
-- 
1.7.7.6

[PATCH 4/4] Test on solib load and unload

[Yao Qi]

This patch is to add a test case to on the performance of GDB handling
load and unload of shared library.

In V4:

 - Handle malloc and dlopen failure,
 - Document test parameters.

In V3, there are some changes,

 - Adapt to perf test framework changes.
 - Measure load and unload separately.

In V2, there are some changes,

 - A new proc gdb_produce_source to produce source files.  I tried to
   move all source file generation code out of solib.exp, but
   compilation step still needs to know the generated file names.  I
   have to hard-code the file names in compilation step, which is not
   good to me, so I give up on this moving.
 - SOLIB_NUMBER -> SOLIB_COUNT
 - New variable SOLIB_DLCLOSE_REVERSED_ORDER to control the order of
   iterating a list of shared libs to dlclose them.
 - New variable GDB_PERFORMANCE to enable these perf test cases.
 - Remove dlsym call in solib.c.
 - Update solib.py for the updated framework.

gdb/testsuite/

	* lib/gdb.exp (gdb_produce_source): New procedure.
	* gdb.perf/solib.c: New.
	* gdb.perf/solib.exp: New.
	* gdb.perf/solib.py: New.
---
 gdb/testsuite/gdb.perf/solib.c   |   83 +++++++++++++++++++++++++++++++++++++
 gdb/testsuite/gdb.perf/solib.exp |   85 ++++++++++++++++++++++++++++++++++++++
 gdb/testsuite/gdb.perf/solib.py  |   73 ++++++++++++++++++++++++++++++++
 gdb/testsuite/lib/gdb.exp        |   16 +++++++
 4 files changed, 257 insertions(+), 0 deletions(-)
 create mode 100644 gdb/testsuite/gdb.perf/solib.c
 create mode 100644 gdb/testsuite/gdb.perf/solib.exp
 create mode 100644 gdb/testsuite/gdb.perf/solib.py

diff --git a/gdb/testsuite/gdb.perf/solib.c b/gdb/testsuite/gdb.perf/solib.c
new file mode 100644
index 0000000..5da944d
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.c
@@ -0,0 +1,83 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright (C) 2013 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __WIN32__
+#include <windows.h>
+#define dlopen(name, mode) LoadLibrary (TEXT (name))
+# define dlsym(handle, func) GetProcAddress (handle, func)
+#define dlclose(handle) FreeLibrary (handle)
+#else
+#include <dlfcn.h>
+#endif
+
+static void **handles;
+
+void
+do_test_load (int number)
+{
+  char libname[40];
+  int i;
+
+  handles = malloc (sizeof (void *) * number);
+  if (handles == NULL)
+    {
+      printf ("ERROR on malloc\n");
+      exit (-1);
+    }
+
+  for (i = 0; i < number; i++)
+    {
+      sprintf (libname, "solib-lib%d", i);
+      handles[i] = dlopen (libname, RTLD_LAZY);
+      if (handles[i] == NULL)
+	{
+	  printf ("ERROR on dlopen %s\n", libname);
+	  exit (-1);
+	}
+    }
+}
+
+void
+do_test_unload (int number)
+{
+  int i;
+
+  /* Unload shared libraries in different orders.  */
+#ifndef SOLIB_DLCLOSE_REVERSED_ORDER
+  for (i = 0; i < number; i++)
+#else
+  for (i = number - 1; i >= 0; i--)
+#endif
+    dlclose (handles[i]);
+
+  free (handles);
+}
+
+static void
+end (void)
+{}
+
+int
+main (void)
+{
+  end ();
+
+  return 0;
+}
diff --git a/gdb/testsuite/gdb.perf/solib.exp b/gdb/testsuite/gdb.perf/solib.exp
new file mode 100644
index 0000000..573a709
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.exp
@@ -0,0 +1,85 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This test case is to test the performance of GDB when it is handling
+# the shared libraries of inferior are loaded and unloaded.
+# There are two parameters in this test:
+#  - SOLIB_COUNT is the number of shared libraries program will load
+#    and unload.
+#  - SOLIB_DLCLOSE_REVERSED_ORDER controls the order of dlclose shared
+#    libraries.  If it is set, program dlclose shared libraries in a
+#    reversed order of loading.
+
+load_lib perftest.exp
+
+if [skip_perf_tests] {
+    return 0
+}
+
+standard_testfile .c
+set executable $testfile
+set expfile $testfile.exp
+
+# make check-perf RUNTESTFLAGS='solib.exp SOLIB_COUNT=1024'
+if ![info exists SOLIB_COUNT] {
+    set SOLIB_COUNT 128
+}
+
+PerfTest::assemble {
+    compile {
+	for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
+
+	    # Produce source files.
+	    set libname "solib-lib$i"
+	    set src [standard_output_file $libname.c]
+	    set exe [standard_output_file $libname]
+
+	    gdb_produce_source $src { "int shr$i (void) {return 0;}" }
+
+	    # Compile.
+	    if { [gdb_compile_shlib $src $exe {debug}] != "" } {
+		untested "Couldn't compile $src."
+		return -1
+	    }
+
+	    # Delete object files to save some space.
+	    file delete [standard_output_file  "solib-lib$i.c.o"]
+	}
+
+	set compile_flags {debug shlib_load}
+	global SOLIB_DLCLOSE_REVERSED_ORDER
+
+	if [info exists SOLIB_DLCLOSE_REVERSED_ORDER] {
+	    lappend compile_flags "additional_flags=-DSOLIB_DLCLOSE_REVERSED_ORDER"
+	}
+
+	if { [gdb_compile "$srcdir/$subdir/$srcfile" ${binfile} executable  $compile_flags] == "" } {
+	    PerfTest::compiled
+	}
+    }
+} {
+   startup_gdb {
+	clean_restart $binfile
+
+	if ![runto_main] {
+	    fail "Can't run to main"
+	    return -1
+	}
+    }
+} {
+    run {
+	gdb_test_no_output "python SolibLoadUnload\($SOLIB_COUNT\).run()"
+    }
+}
diff --git a/gdb/testsuite/gdb.perf/solib.py b/gdb/testsuite/gdb.perf/solib.py
new file mode 100644
index 0000000..91393b1
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.py
@@ -0,0 +1,73 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This test case is to test the speed of GDB when it is handling the
+# shared libraries of inferior are loaded and unloaded.
+
+from perftest import perftest
+from perftest import measure
+
+class SolibLoadUnload1(perftest.TestCaseWithBasicMeasurements):
+    def __init__(self, solib_count, measure_load):
+        if measure_load:
+            name = "solib_load"
+        else:
+            name = "solib_unload"
+        # We want to measure time in this test.
+        super (SolibLoadUnload1, self).__init__ (name)
+        self.solib_count = solib_count
+        self.measure_load = measure_load
+
+    def warm_up(self):
+        do_test_load = "call do_test_load (%d)" % self.solib_count
+        do_test_unload = "call do_test_unload (%d)" % self.solib_count
+        gdb.execute(do_test_load)
+        gdb.execute(do_test_unload)
+
+    def execute_test(self):
+        num = self.solib_count
+        iteration = 5;
+
+        while num > 0 and iteration > 0:
+            # Do inferior calls to do_test_load and do_test_unload in pairs,
+            # but measure differently.
+            if self.measure_load:
+                do_test_load = "call do_test_load (%d)" % num
+                func = lambda: gdb.execute (do_test_load)
+
+                self.measure.measure(func, num)
+
+                do_test_unload = "call do_test_unload (%d)" % num
+                gdb.execute (do_test_unload)
+
+            else:
+                do_test_load = "call do_test_load (%d)" % num
+                gdb.execute (do_test_load)
+
+                do_test_unload = "call do_test_unload (%d)" % num
+                func = lambda: gdb.execute (do_test_unload)
+
+                self.measure.measure(func, num)
+
+            num = num / 2
+            iteration -= 1
+
+class SolibLoadUnload(object):
+    def __init__(self, solib_count):
+        self.solib_count = solib_count;
+
+    def run(self):
+        SolibLoadUnload1(self.solib_count, True).run()
+        SolibLoadUnload1(self.solib_count, False).run()
diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index 3df1c8f..5d2ae50 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -1835,6 +1835,22 @@ gdb_caching_proc is_elf_target {
     return 1
 }
 
+# Produce source file NAME and write SOURCES into it.
+
+proc gdb_produce_source { name sources } {
+    set index 0
+    set f [open $name "w"]
+
+    while { ${index} < [llength ${sources}] } {
+	set line [lindex ${sources} ${index}]
+	set index [expr ${index} + 1]
+
+	set line [uplevel list $line]
+	puts $f $line
+    }
+    close $f
+}
+
 # Return 1 if target is ILP32.
 # This cannot be decided simply from looking at the target string,
 # as it might depend on externally passed compiler options like -m64.
-- 
1.7.7.6

[PATCH 0/4 V4] GDB Performance testing

[Yao Qi]

Hi,
Here is the V4 of GDB performance testing.  Doug approved V3 except
patch 2/4.  V4 addresses review comments to V3, including Pedro's
suggestion on renaming parameters.

The changes in V4 can be found in each patch.

The basic usages of performance testing are,

$ make check-perf
$ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver solib.exp"
$ make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=512"

We can skip compilation step like,

$ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFTEST_MODE=run'

or only compile test cases,

$ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFTEST_MODE=compile'

in default, GDB_PERFTEST_MODE=both.

If the perf test case takes much time, we can specify timeout by
GDB_PERFTEST_TIMEOUT=1000.

$ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver GDB_PERFTEST_TIMEOUT=4000 solib.exp"

*** BLURB HERE ***

Yao Qi (4):
  New make target 'check-perf' and new dir gdb.perf
  Perf test framework
  Mention perf test in testsuite/README
  Test on solib load and unload

 gdb/Makefile.in                                   |    8 +
 gdb/testsuite/Makefile.in                         |    4 +
 gdb/testsuite/README                              |   27 ++++
 gdb/testsuite/configure                           |    3 +-
 gdb/testsuite/configure.ac                        |    2 +-
 gdb/testsuite/gdb.perf/Makefile.in                |   15 ++
 gdb/testsuite/gdb.perf/lib/perftest/__init__.py   |   17 +++
 gdb/testsuite/gdb.perf/lib/perftest/measure.py    |  146 ++++++++++++++++++++
 gdb/testsuite/gdb.perf/lib/perftest/perftest.py   |   73 ++++++++++
 gdb/testsuite/gdb.perf/lib/perftest/reporter.py   |   64 +++++++++
 gdb/testsuite/gdb.perf/lib/perftest/testresult.py |   57 ++++++++
 gdb/testsuite/gdb.perf/solib.c                    |   83 ++++++++++++
 gdb/testsuite/gdb.perf/solib.exp                  |   85 ++++++++++++
 gdb/testsuite/gdb.perf/solib.py                   |   73 ++++++++++
 gdb/testsuite/lib/gdb.exp                         |   16 +++
 gdb/testsuite/lib/perftest.exp                    |  148 +++++++++++++++++++++
 16 files changed, 819 insertions(+), 2 deletions(-)
 create mode 100644 gdb/testsuite/gdb.perf/Makefile.in
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
 create mode 100644 gdb/testsuite/gdb.perf/solib.c
 create mode 100644 gdb/testsuite/gdb.perf/solib.exp
 create mode 100644 gdb/testsuite/gdb.perf/solib.py
 create mode 100644 gdb/testsuite/lib/perftest.exp

-- 
1.7.7.6

[PATCH 3/4] Mention perf test in testsuite/README

[Yao Qi]

gdb/testsuite:

2013-11-01  Yao Qi  <yao@codesourcery.com>

	* README: Mention performance tests.
---
 gdb/testsuite/README |   27 +++++++++++++++++++++++++++
 1 files changed, 27 insertions(+), 0 deletions(-)

diff --git a/gdb/testsuite/README b/gdb/testsuite/README
index ec91b14..c17bd34 100644
--- a/gdb/testsuite/README
+++ b/gdb/testsuite/README
@@ -25,6 +25,29 @@ The second is to cd to the testsuite directory and invoke the DejaGnu
 (The `site.exp' file contains a handful of useful variables like host
 and target triplets, and pathnames.)
 
+Running the Performance Tests
+*****************************
+
+GDB Testsuite includes performance test cases, which are not run together
+with other test cases, because performance test cases are slow and need
+a quiet system.  There are two ways to run the performance test cases.
+The first is to do `make check-perf' in the main build directory:
+
+	make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=8"
+
+The second is to cd to the testsuite directory and invoke the DejaGnu
+`runtest' command directly.
+
+	cd testsuite
+	make site.exp
+	runtest GDB_PERFTEST_MODE=both GDB_PERFTEST_TIMEOUT=4000 --directory=gdb.perf solib.exp SOLIB_COUNT=8
+
+Only "compile", "run" and "both" are valid to GDB_PERFTEST_MODE.  They
+stand for "compile tests only", "run tests only", and "compile and run
+tests" respectively.  "both" is the default.  GDB_PERFTEST_TIMEOUT
+specify the timeout, which is 3000 in default.  The result of
+performance test is appended in `testsuite/perftest.log'.
+
 Testsuite Parameters
 ********************
 
@@ -315,6 +338,10 @@ Tests that exercise a specific GDB subsystem in more depth.  For
 instance, gdb.disasm exercises various disassemblers, while
 gdb.stabs tests pathways through the stabs symbol reader.
 
+gdb.perf
+
+GDB performance tests.
+
 Writing Tests
 *************
 
-- 
1.7.7.6

Re: [PATCH 2/4] Perf test framework

[Doug Evans]

On Fri, Nov 1, 2013 at 12:34 AM, Yao Qi <yao@codesourcery.com> wrote:
> This patch adds a basic framework to do performance testing for GDB.
> perftest.py is about the test case, testresult.py is about test
> results, and how are they saved.  reporter.py is about how results
> are reported (in what format).  measure.py is about measuring the
> execution of tests by a collection of measurements.
>
> In V4:
>
>  - Rename MeasurementCPUTime to MeasurementCpuTime,
>  - Add 'pass' in empty method,
>  - Simplify string comparison in perftest.exp.
>  - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename
>    GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT.
>
> In V3, there are some changes,
>
>  - Add wall time measurement, cpu time measurement and vmsize
>    measurement.
>  - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements,
>    which measures cpu time, wall time, and memory (vmsize).
>  - GDB_PERFORMANCE=run|compile|both to control the mode of perf
>    testing.
>  - New GDB_PERFORMANCE_TIMEOUT to specify the timeout.
>  - Split proc prepare to proc compile and startup.
>  - Disable GC while doing measurements.
>
> In V2, there are several changes to address Doug and Sanimir's
> comments.
>
>  - Add copyright header and docstring in perftest/__init__.py
>  - Remove config.py.
>  - Fix docstring format.
>  - Rename classes "SingleVariable" to "SingleStatistic".
>  - Don't extend gdb.Function in class TestCase.  Add a new method run
>    to run the test case so that we can pass parameters to test.
>  - Allow to customize whether to warm up and to append test log.
>  - Move time measurement into test harness.  Add a new class
>    Measurement for a specific measurement and a new class Measure to
>    measure them for a given test case.
>  - A new class ResultFactory to create instances of TestResult.
>  - New file lib/perftest.exp, which is to do some preparations and
>    cleanups to simplify each *.exp file.
>  - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set.
>
> gdb/testsuite/
>
>         * lib/perftest.exp: New.
>         * gdb.perf/lib/perftest/__init__.py: New.
>         * gdb.perf/lib/perftest/measure.py: New.
>         * gdb.perf/lib/perftest/perftest.py: New.
>         * gdb.perf/lib/perftest/reporter.py: New.
>         * gdb.perf/lib/perftest/testresult.py: New.

Ok by me.
[Might want to give it a day or two to see if anyone else has any
final comments.]

Thanks for the perseverance!

Re: [PATCH 4/4] Test on solib load and unload

[Tom Tromey]

>>>>> "Yao" == Yao Qi <yao@codesourcery.com> writes:

Yao> This patch is to add a test case to on the performance of GDB handling
Yao> load and unload of shared library.

A few Tcl nits on this patch...

Yao> +	    gdb_produce_source $src { "int shr$i (void) {return 0;}" }

This bit is weird.  Combined with the definition of gdb_produce_source:

Yao> +proc gdb_produce_source { name sources } {
Yao> +    set index 0
Yao> +    set f [open $name "w"]
Yao> +
Yao> +    while { ${index} < [llength ${sources}] } {
Yao> +	set line [lindex ${sources} ${index}]
Yao> +	set index [expr ${index} + 1]
Yao> +
Yao> +	set line [uplevel list $line]
Yao> +	puts $f $line
Yao> +    }
Yao> +    close $f
Yao> +}

I don't see why this "uplevel list" stuff is needed.
Just have the caller pass in the text to write to the file.
The substitutions can be done there.

There also doesn't seem to be a need for "sources" to be a list.
Just make it one big string and let the caller work it out.


Yao> +	    # Compile.
Yao> +	    if { [gdb_compile_shlib $src $exe {debug}] != "" } {
Yao> +		untested "Couldn't compile $src."
Yao> +		return -1

This returns a value but I think the caller doesn't use it.
This isn't a bug, but it does mean some head-scratching later on, and of
course this will inevitably be copied into all the other perf tests...

That said I think this probably *should* return a value and I'll comment
on the other patch.

Tom

Re: [PATCH 2/4] Perf test framework

[Tom Tromey]

>>>>> "Yao" == Yao Qi <yao@codesourcery.com> writes:

Yao> +    # Compile source files of test case.  BODY is the tcl code to do
Yao> +    # actual compilation and it should invoke 'PerfTest::compiled' if
Yao> +    # compilation is successful.

Why not just have the body return success or failure and avoid having to
call another method?

Yao> +    # The top-level interface to PerfTest.
Yao> +    # COMPILE is the tcl code to generate and compile source files.
Yao> +    # STARTUP is the tcl code to start up GDB.
Yao> +    # RUN is the tcl code to drive GDB to do some operations.
Yao> +    proc assemble {compile startup run} {

This approach ends up using a lot of indentation, it seems to me.
Slimmer approaches seem possible without losing anything.

E.g., in patch #4 I see:

+PerfTest::assemble {
+    compile {
+	for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
+


But it seems to me that the "compile" part there is strictly redundant.
We already know that this is the "compile" branch, so why not have
"assemble" supply that?

Then the above would read:

    PerfTest::assemble {
        for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
    ...

Tom

Re: [PATCH 2/4] Perf test framework

[Yao Qi]

On 11/02/2013 04:46 AM, Tom Tromey wrote:
> Yao> +    # Compile source files of test case.  BODY is the tcl code to do
> Yao> +    # actual compilation and it should invoke 'PerfTest::compiled' if
> Yao> +    # compilation is successful.
> 
> Why not just have the body return success or failure and avoid having to
> call another method?
> 

Yeah, we can do that.  Method PerfTest::compiled and variable
compiled_ok can be removed.

> Yao> +    # The top-level interface to PerfTest.
> Yao> +    # COMPILE is the tcl code to generate and compile source files.
> Yao> +    # STARTUP is the tcl code to start up GDB.
> Yao> +    # RUN is the tcl code to drive GDB to do some operations.
> Yao> +    proc assemble {compile startup run} {
> 
> This approach ends up using a lot of indentation, it seems to me.
> Slimmer approaches seem possible without losing anything.
> 
> E.g., in patch #4 I see:
> 
> +PerfTest::assemble {
> +    compile {
> +	for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
> +
> 
> 
> But it seems to me that the "compile" part there is strictly redundant.
> We already know that this is the "compile" branch, so why not have
> "assemble" supply that?
> 
> Then the above would read:
> 
>      PerfTest::assemble {
>          for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
>      ...

OK, that will simplify perftest.exp and test cases.

-- 

gdb/testsuite/

	* lib/perftest.exp: New.
	* gdb.perf/lib/perftest/__init__.py: New.
	* gdb.perf/lib/perftest/measure.py: New.
	* gdb.perf/lib/perftest/perftest.py: New.
	* gdb.perf/lib/perftest/reporter.py: New.
	* gdb.perf/lib/perftest/testresult.py: New.
---
 gdb/testsuite/gdb.perf/lib/perftest/__init__.py   |   17 +++
 gdb/testsuite/gdb.perf/lib/perftest/measure.py    |  146 +++++++++++++++++++++
 gdb/testsuite/gdb.perf/lib/perftest/perftest.py   |   73 ++++++++++
 gdb/testsuite/gdb.perf/lib/perftest/reporter.py   |   64 +++++++++
 gdb/testsuite/gdb.perf/lib/perftest/testresult.py |   57 ++++++++
 gdb/testsuite/lib/perftest.exp                    |  125 ++++++++++++++++++
 6 files changed, 482 insertions(+), 0 deletions(-)
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/__init__.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/measure.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/perftest.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/reporter.py
 create mode 100644 gdb/testsuite/gdb.perf/lib/perftest/testresult.py
 create mode 100644 gdb/testsuite/lib/perftest.exp

diff --git a/gdb/testsuite/gdb.perf/lib/perftest/__init__.py b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
new file mode 100644
index 0000000..7739a3e
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/__init__.py
@@ -0,0 +1,17 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+GDB performance testing framework.
+"""
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/measure.py b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
new file mode 100644
index 0000000..b7cf4e7
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/measure.py
@@ -0,0 +1,146 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import time
+import os
+import gc
+
+class Measure(object):
+    """A class that measure and collect the interesting data for a given testcase.
+
+    An instance of Measure has a collection of measurements, and each
+    of them is to measure a given aspect, such as time and memory.
+    """
+
+    def __init__(self, measurements):
+        """Constructor of measure.
+
+        measurements is a collection of Measurement objects.
+        """
+
+        self.measurements = measurements
+
+    def measure(self, func, id):
+        """Measure the operations done by func with a collection of measurements."""
+        # Enable GC, force GC and disable GC before running test in order to reduce
+        # the interference from GC.
+        gc.enable()
+        gc.collect()
+        gc.disable()
+
+        for m in self.measurements:
+            m.start(id)
+
+        func()
+
+        for m in self.measurements:
+            m.stop(id)
+
+        gc.enable()
+
+    def report(self, reporter, name):
+        """Report the measured results."""
+        for m in self.measurements:
+            m.report(reporter, name)
+
+class Measurement(object):
+    """A measurement for a certain aspect."""
+
+    def __init__(self, name, result):
+        """Constructor of Measurement.
+
+        Attribute result is the TestResult associated with measurement.
+        """
+        self.name = name;
+        self.result = result
+
+    def start(self, id):
+        """Abstract method to start the measurement."""
+        raise NotImplementedError("Abstract Method:start")
+
+    def stop(self, id):
+        """Abstract method to stop the measurement.
+
+        When the measurement is stopped, we've got something, and
+        record them in result.
+        """
+        raise NotImplementedError("Abstract Method:stop.")
+
+    def report(self, reporter, name):
+        """Report the measured data by argument reporter."""
+        self.result.report(reporter, name + " " + self.name)
+
+class MeasurementCpuTime(Measurement):
+    """Measurement on CPU time."""
+    # On UNIX, time.clock() measures the amount of CPU time that has
+    # been used by the current process.  On Windows it will measure
+    # wall-clock seconds elapsed since the first call to the function.
+    # Something other than time.clock() should be used to measure CPU
+    # time on Windows.
+
+    def __init__(self, result):
+        super(MeasurementCpuTime, self).__init__("cpu_time", result)
+        self.start_time = 0
+
+    def start(self, id):
+        self.start_time = time.clock()
+
+    def stop(self, id):
+        if os.name == 'nt':
+            cpu_time = 0
+        else:
+            cpu_time = time.clock() - self.start_time
+        self.result.record (id, cpu_time)
+
+class MeasurementWallTime(Measurement):
+    """Measurement on Wall time."""
+
+    def __init__(self, result):
+        super(MeasurementWallTime, self).__init__("wall_time", result)
+        self.start_time = 0
+
+    def start(self, id):
+        self.start_time = time.time()
+
+    def stop(self, id):
+        wall_time = time.time() - self.start_time
+        self.result.record (id, wall_time)
+
+class MeasurementVmSize(Measurement):
+    """Measurement on memory usage represented by VmSize."""
+
+    def __init__(self, result):
+        super(MeasurementVmSize, self).__init__("vmsize", result)
+
+    def _compute_process_memory_usage(self, key):
+        file_path = "/proc/%d/status" % os.getpid()
+        try:
+            t = open(file_path)
+            v = t.read()
+            t.close()
+        except:
+            return 0
+        i = v.index(key)
+        v = v[i:].split(None, 3)
+        if len(v) < 3:
+            return 0
+        return int(v[1])
+
+    def start(self, id):
+        pass
+
+    def stop(self, id):
+        memory_used = self._compute_process_memory_usage("VmSize:")
+        self.result.record (id, memory_used)
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/perftest.py b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
new file mode 100644
index 0000000..e398230
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/perftest.py
@@ -0,0 +1,73 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import testresult
+import reporter
+from measure import Measure
+from measure import MeasurementCpuTime
+from measure import MeasurementWallTime
+from measure import MeasurementVmSize
+
+class TestCase(object):
+    """Base class of all performance testing cases.
+
+    Each sub-class should override methods execute_test, in which
+    several GDB operations are performed and measured by attribute
+    measure.  Sub-class can also override method warm_up optionally
+    if the test case needs warm up.
+    """
+
+    def __init__(self, name, measure):
+        """Constructor of TestCase.
+
+        Construct an instance of TestCase with a name and a measure
+        which is to measure the test by several different measurements.
+        """
+
+        self.name = name
+        self.measure = measure
+
+    def execute_test(self):
+        """Abstract method to do the actual tests."""
+        raise NotImplementedError("Abstract Method.")
+
+    def warm_up(self):
+        """Do some operations to warm up the environment."""
+        pass
+
+    def run(self, warm_up=True, append=True):
+        """Run this test case.
+
+        It is a template method to invoke method warm_up,
+        execute_test, and finally report the measured results.
+        If parameter warm_up is True, run method warm_up.  If parameter
+        append is True, the test result will be appended instead of
+        overwritten.
+        """
+        if warm_up:
+            self.warm_up()
+
+        self.execute_test()
+        self.measure.report(reporter.TextReporter(append), self.name)
+
+class TestCaseWithBasicMeasurements(TestCase):
+    """Test case measuring CPU time, wall time and memory usage."""
+
+    def __init__(self, name):
+        result_factory = testresult.SingleStatisticResultFactory()
+        measurements = [MeasurementCpuTime(result_factory.create_result()),
+                        MeasurementWallTime(result_factory.create_result()),
+                        MeasurementVmSize(result_factory.create_result())]
+        super (TestCaseWithBasicMeasurements, self).__init__ (name, Measure(measurements))
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/reporter.py b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
new file mode 100644
index 0000000..a902e4b
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/reporter.py
@@ -0,0 +1,64 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+class Reporter(object):
+    """Base class of reporter to report test results in a certain format.
+
+    Subclass, which is specific to a report format, should overwrite
+    methods report, start and end.
+    """
+
+    def __init__(self, append):
+        """Constructor of Reporter.
+
+        attribute append is used to determine whether to append or
+        overwrite log file.
+        """
+        self.append = append
+
+    def report(self, *args):
+        raise NotImplementedError("Abstract Method:report.")
+
+    def start(self):
+        """Invoked when reporting is started."""
+        raise NotImplementedError("Abstract Method:start.")
+
+    def end(self):
+        """Invoked when reporting is done.
+
+        It must be overridden to do some cleanups, such as closing file
+        descriptors.
+        """
+        raise NotImplementedError("Abstract Method:end.")
+
+class TextReporter(Reporter):
+    """Report results in a plain text file 'perftest.log'."""
+
+    def __init__(self, append):
+        super (TextReporter, self).__init__(Reporter(append))
+        self.txt_log = None
+
+    def report(self, *args):
+        self.txt_log.write(' '.join(str(arg) for arg in args))
+        self.txt_log.write('\n')
+
+    def start(self):
+        if self.append:
+            self.txt_log = open ("perftest.log", 'a+');
+        else:
+            self.txt_log = open ("perftest.log", 'w');
+
+    def end(self):
+        self.txt_log.close ()
diff --git a/gdb/testsuite/gdb.perf/lib/perftest/testresult.py b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
new file mode 100644
index 0000000..e571f12
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/lib/perftest/testresult.py
@@ -0,0 +1,57 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+class TestResult(object):
+    """Base class to record and report test results.
+
+    Method record is to record the results of test case, and report
+    method is to report the recorded results by a given reporter.
+    """
+
+    def record(self, parameter, result):
+        raise NotImplementedError("Abstract Method:record.")
+
+    def report(self, reporter, name):
+        """Report the test results by reporter."""
+        raise NotImplementedError("Abstract Method:report.")
+
+class SingleStatisticTestResult(TestResult):
+    """Test results for the test case with a single statistic."""
+
+    def __init__(self):
+        super (SingleStatisticTestResult, self).__init__ ()
+        self.results = dict ()
+
+    def record(self, parameter, result):
+        self.results[parameter] = result
+
+    def report(self, reporter, name):
+        reporter.start()
+        for key in sorted(self.results.iterkeys()):
+            reporter.report(name, key, self.results[key])
+        reporter.end()
+
+class ResultFactory(object):
+    """A factory to create an instance of TestResult."""
+
+    def create_result(self):
+        """Create an instance of TestResult."""
+        raise NotImplementedError("Abstract Method:create_result.")
+
+class SingleStatisticResultFactory(ResultFactory):
+    """A factory to create an instance of SingleStatisticTestResult."""
+
+    def create_result(self):
+        return SingleStatisticTestResult()
diff --git a/gdb/testsuite/lib/perftest.exp b/gdb/testsuite/lib/perftest.exp
new file mode 100644
index 0000000..988e4c2
--- /dev/null
+++ b/gdb/testsuite/lib/perftest.exp
@@ -0,0 +1,125 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace eval PerfTest {
+    # The name of python file on build.
+    variable remote_python_file
+
+    # A private method to set up GDB for performance testing.
+    proc _setup_perftest {} {
+	variable remote_python_file
+	global srcdir subdir testfile
+
+	set remote_python_file [gdb_remote_download host ${srcdir}/${subdir}/${testfile}.py]
+
+	# Set sys.path for module perftest.
+	gdb_test_no_output "python import os, sys"
+	gdb_test_no_output "python sys.path.insert\(0, os.path.abspath\(\"${srcdir}/${subdir}/lib\"\)\)"
+	gdb_test_no_output "python exec (open ('${remote_python_file}').read ())"
+    }
+
+    # A private method to do some cleanups when performance test is
+    # finished.
+    proc _teardown_perftest {} {
+	variable remote_python_file
+
+	remote_file host delete $remote_python_file
+    }
+
+    # Compile source files of test case.  BODY is the tcl code to do
+    # actual compilation.  Return zero if compilation is successful,
+    # otherwise return non-zero.
+    proc compile {body} {
+	global GDB_PERFTEST_MODE
+
+	if { [info exists GDB_PERFTEST_MODE]
+	     && [string compare $GDB_PERFTEST_MODE "run"] } {
+	    return [uplevel 2 $body]
+	}
+
+	return 0
+    }
+
+    # Start up GDB.
+    proc startup_gdb {body} {
+	uplevel 2 $body
+    }
+
+    # Run the performance test.
+    proc run {body} {
+	global timeout
+	global GDB_PERFTEST_TIMEOUT
+
+	set oldtimeout $timeout
+	if { [info exists GDB_PERFTEST_TIMEOUT] } {
+	    set timeout $GDB_PERFTEST_TIMEOUT
+	} else {
+	    set timeout 3000
+	}
+	uplevel 2 $body
+
+	set timeout $oldtimeout
+    }
+
+    # The top-level interface to PerfTest.
+    # COMPILE is the tcl code to generate and compile source files.
+    # Return zero if compilation is successful,  otherwise return
+    # non-zero.
+    # STARTUP is the tcl code to start up GDB.
+    # RUN is the tcl code to drive GDB to do some operations.
+    proc assemble {compile startup run} {
+	global GDB_PERFTEST_MODE
+
+	if { [eval compile {$compile}] } {
+	    untested "Could not compile source files."
+	    return
+	}
+
+	# Don't execute the run if GDB_PERFTEST_MODE=compile.
+	if { [info exists GDB_PERFTEST_MODE]
+	     && [string compare $GDB_PERFTEST_MODE "compile"] == 0} {
+	    return
+	}
+
+	eval $startup
+
+	_setup_perftest
+
+	eval run {$run}
+
+	_teardown_perftest
+    }
+}
+
+# Return true if performance tests are skipped.
+
+proc skip_perf_tests { } {
+    global GDB_PERFTEST_MODE
+
+    if [info exists GDB_PERFTEST_MODE] {
+
+	if { "$GDB_PERFTEST_MODE" != "compile"
+	     && "$GDB_PERFTEST_MODE" != "run"
+	     && "$GDB_PERFTEST_MODE" != "both" } {
+	    # GDB_PERFTEST_MODE=compile|run|both is allowed.
+	    unsupported "Unknown value of GDB_PERFTEST_MODE."
+	    return 1
+	}
+
+	return 0
+    }
+
+    return 1
+}
-- 
1.7.7.6

Re: [PATCH 4/4] Test on solib load and unload

[Yao Qi]

On 11/02/2013 04:35 AM, Tom Tromey wrote:
> Yao> +proc gdb_produce_source { name sources } {
> Yao> +    set index 0
> Yao> +    set f [open $name "w"]
> Yao> +
> Yao> +    while { ${index} < [llength ${sources}] } {
> Yao> +	set line [lindex ${sources} ${index}]
> Yao> +	set index [expr ${index} + 1]
> Yao> +
> Yao> +	set line [uplevel list $line]
> Yao> +	puts $f $line
> Yao> +    }
> Yao> +    close $f
> Yao> +}
> 
> I don't see why this "uplevel list" stuff is needed.
> Just have the caller pass in the text to write to the file.
> The substitutions can be done there.

Right, I recall that literal "$i" was written into the source file and
it makes me use "uplevel".  However, "uplevel list" is unnecessary
now.  Fixed.

> 
> There also doesn't seem to be a need for "sources" to be a list.
> Just make it one big string and let the caller work it out.
> 

Using a list makes caller code cleaner if multiple lines are written.
I am fine to use one big string here.

> 
> Yao> +	    # Compile.
> Yao> +	    if { [gdb_compile_shlib $src $exe {debug}] != "" } {
> Yao> +		untested "Couldn't compile $src."
> Yao> +		return -1
> 
> This returns a value but I think the caller doesn't use it.
> This isn't a bug, but it does mean some head-scratching later on, and of
> course this will inevitably be copied into all the other perf tests...

Updated perf test framework in patch 2/4 uses the return value.
-- 
Yao (齐尧)

gdb/testsuite/

	* lib/gdb.exp (gdb_produce_source): New procedure.
	* gdb.perf/solib.c: New.
	* gdb.perf/solib.exp: New.
	* gdb.perf/solib.py: New.
---
 gdb/testsuite/gdb.perf/solib.c   |   83 ++++++++++++++++++++++++++++++++++++
 gdb/testsuite/gdb.perf/solib.exp |   87 ++++++++++++++++++++++++++++++++++++++
 gdb/testsuite/gdb.perf/solib.py  |   73 +++++++++++++++++++++++++++++++
 gdb/testsuite/lib/gdb.exp        |   10 ++++
 4 files changed, 253 insertions(+), 0 deletions(-)
 create mode 100644 gdb/testsuite/gdb.perf/solib.c
 create mode 100644 gdb/testsuite/gdb.perf/solib.exp
 create mode 100644 gdb/testsuite/gdb.perf/solib.py

diff --git a/gdb/testsuite/gdb.perf/solib.c b/gdb/testsuite/gdb.perf/solib.c
new file mode 100644
index 0000000..5da944d
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.c
@@ -0,0 +1,83 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright (C) 2013 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __WIN32__
+#include <windows.h>
+#define dlopen(name, mode) LoadLibrary (TEXT (name))
+# define dlsym(handle, func) GetProcAddress (handle, func)
+#define dlclose(handle) FreeLibrary (handle)
+#else
+#include <dlfcn.h>
+#endif
+
+static void **handles;
+
+void
+do_test_load (int number)
+{
+  char libname[40];
+  int i;
+
+  handles = malloc (sizeof (void *) * number);
+  if (handles == NULL)
+    {
+      printf ("ERROR on malloc\n");
+      exit (-1);
+    }
+
+  for (i = 0; i < number; i++)
+    {
+      sprintf (libname, "solib-lib%d", i);
+      handles[i] = dlopen (libname, RTLD_LAZY);
+      if (handles[i] == NULL)
+	{
+	  printf ("ERROR on dlopen %s\n", libname);
+	  exit (-1);
+	}
+    }
+}
+
+void
+do_test_unload (int number)
+{
+  int i;
+
+  /* Unload shared libraries in different orders.  */
+#ifndef SOLIB_DLCLOSE_REVERSED_ORDER
+  for (i = 0; i < number; i++)
+#else
+  for (i = number - 1; i >= 0; i--)
+#endif
+    dlclose (handles[i]);
+
+  free (handles);
+}
+
+static void
+end (void)
+{}
+
+int
+main (void)
+{
+  end ();
+
+  return 0;
+}
diff --git a/gdb/testsuite/gdb.perf/solib.exp b/gdb/testsuite/gdb.perf/solib.exp
new file mode 100644
index 0000000..c1b00b0
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.exp
@@ -0,0 +1,87 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This test case is to test the performance of GDB when it is handling
+# the shared libraries of inferior are loaded and unloaded.
+# There are two parameters in this test:
+#  - SOLIB_COUNT is the number of shared libraries program will load
+#    and unload
+#  - SOLIB_DLCLOSE_REVERSED_ORDER controls the order of dlclose shared
+#    libraries.  If it is set, program dlclose shared libraries in a
+#    reversed order of loading.
+
+load_lib perftest.exp
+
+if [skip_perf_tests] {
+    return 0
+}
+
+standard_testfile .c
+set executable $testfile
+set expfile $testfile.exp
+
+# make check-perf RUNTESTFLAGS='solib.exp SOLIB_COUNT=1024'
+if ![info exists SOLIB_COUNT] {
+    set SOLIB_COUNT 128
+}
+
+PerfTest::assemble {
+    global SOLIB_COUNT
+    global srcdir subdir srcfile binfile
+
+    for {set i 0} {$i < $SOLIB_COUNT} {incr i} {
+
+	# Produce source files.
+	set libname "solib-lib$i"
+	set src [standard_output_file $libname.c]
+	set exe [standard_output_file $libname]
+
+	gdb_produce_source $src "int shr$i (void) {return 0;}"
+
+	# Compile.
+	if { [gdb_compile_shlib $src $exe {debug}] != "" } {
+	    return -1
+	}
+
+	# Delete object files to save some space.
+	file delete [standard_output_file  "solib-lib$i.c.o"]
+    }
+
+    set compile_flags {debug shlib_load}
+    global SOLIB_DLCLOSE_REVERSED_ORDER
+
+    if [info exists SOLIB_DLCLOSE_REVERSED_ORDER] {
+	lappend compile_flags "additional_flags=-DSOLIB_DLCLOSE_REVERSED_ORDER"
+    }
+
+    if { [gdb_compile "$srcdir/$subdir/$srcfile" ${binfile} executable  $compile_flags] != "" } {
+	return -1
+    }
+
+    return 0
+} {
+    global binfile
+
+    clean_restart $binfile
+
+    if ![runto_main] {
+	fail "Can't run to main"
+	return -1
+    }
+} {
+    global SOLIB_COUNT
+
+    gdb_test_no_output "python SolibLoadUnload\($SOLIB_COUNT\).run()"
+}
diff --git a/gdb/testsuite/gdb.perf/solib.py b/gdb/testsuite/gdb.perf/solib.py
new file mode 100644
index 0000000..91393b1
--- /dev/null
+++ b/gdb/testsuite/gdb.perf/solib.py
@@ -0,0 +1,73 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This test case is to test the speed of GDB when it is handling the
+# shared libraries of inferior are loaded and unloaded.
+
+from perftest import perftest
+from perftest import measure
+
+class SolibLoadUnload1(perftest.TestCaseWithBasicMeasurements):
+    def __init__(self, solib_count, measure_load):
+        if measure_load:
+            name = "solib_load"
+        else:
+            name = "solib_unload"
+        # We want to measure time in this test.
+        super (SolibLoadUnload1, self).__init__ (name)
+        self.solib_count = solib_count
+        self.measure_load = measure_load
+
+    def warm_up(self):
+        do_test_load = "call do_test_load (%d)" % self.solib_count
+        do_test_unload = "call do_test_unload (%d)" % self.solib_count
+        gdb.execute(do_test_load)
+        gdb.execute(do_test_unload)
+
+    def execute_test(self):
+        num = self.solib_count
+        iteration = 5;
+
+        while num > 0 and iteration > 0:
+            # Do inferior calls to do_test_load and do_test_unload in pairs,
+            # but measure differently.
+            if self.measure_load:
+                do_test_load = "call do_test_load (%d)" % num
+                func = lambda: gdb.execute (do_test_load)
+
+                self.measure.measure(func, num)
+
+                do_test_unload = "call do_test_unload (%d)" % num
+                gdb.execute (do_test_unload)
+
+            else:
+                do_test_load = "call do_test_load (%d)" % num
+                gdb.execute (do_test_load)
+
+                do_test_unload = "call do_test_unload (%d)" % num
+                func = lambda: gdb.execute (do_test_unload)
+
+                self.measure.measure(func, num)
+
+            num = num / 2
+            iteration -= 1
+
+class SolibLoadUnload(object):
+    def __init__(self, solib_count):
+        self.solib_count = solib_count;
+
+    def run(self):
+        SolibLoadUnload1(self.solib_count, True).run()
+        SolibLoadUnload1(self.solib_count, False).run()
diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
index 3df1c8f..a3da473 100644
--- a/gdb/testsuite/lib/gdb.exp
+++ b/gdb/testsuite/lib/gdb.exp
@@ -1835,6 +1835,16 @@ gdb_caching_proc is_elf_target {
     return 1
 }
 
+# Produce source file NAME and write SOURCES into it.
+
+proc gdb_produce_source { name sources } {
+    set index 0
+    set f [open $name "w"]
+
+    puts $f $sources
+    close $f
+}
+
 # Return 1 if target is ILP32.
 # This cannot be decided simply from looking at the target string,
 # as it might depend on externally passed compiler options like -m64.
-- 
1.7.7.6

Re: [PATCH 2/4] Perf test framework

[Tom Tromey]

>>>>> "Yao" == Yao Qi <yao@codesourcery.com> writes:

Yao> 	* lib/perftest.exp: New.
Yao> 	* gdb.perf/lib/perftest/__init__.py: New.
Yao> 	* gdb.perf/lib/perftest/measure.py: New.
Yao> 	* gdb.perf/lib/perftest/perftest.py: New.
Yao> 	* gdb.perf/lib/perftest/reporter.py: New.
Yao> 	* gdb.perf/lib/perftest/testresult.py: New.

Thanks for making that change, Yao.

One more little nit:

Yao> +	if { "$GDB_PERFTEST_MODE" != "compile"
Yao> +	     && "$GDB_PERFTEST_MODE" != "run"
Yao> +	     && "$GDB_PERFTEST_MODE" != "both" } {
Yao> +	    # GDB_PERFTEST_MODE=compile|run|both is allowed.
Yao> +	    unsupported "Unknown value of GDB_PERFTEST_MODE."
Yao> +	    return 1

This is basically an internal error.
Calling "error" and having a backtrace in the log seems better to me.

Tom

Re: [PATCH 4/4] Test on solib load and unload

[Tom Tromey]

>>>>> "Yao" == Yao Qi <yao@codesourcery.com> writes:

Yao> Using a list makes caller code cleaner if multiple lines are written.
Yao> I am fine to use one big string here.

You can always join a list in the caller:

   set list [however-you-make-it]
   gdb_produce_source filename [join $list \n]

Yao> 	* lib/gdb.exp (gdb_produce_source): New procedure.
Yao> 	* gdb.perf/solib.c: New.
Yao> 	* gdb.perf/solib.exp: New.
Yao> 	* gdb.perf/solib.py: New.

Looks good to me, thanks.

Tom

Re: [PATCH 2/4] Perf test framework

[Yao Qi]

On 11/05/2013 12:24 AM, Tom Tromey wrote:
> Yao> +	if { "$GDB_PERFTEST_MODE" != "compile"
> Yao> +	     && "$GDB_PERFTEST_MODE" != "run"
> Yao> +	     && "$GDB_PERFTEST_MODE" != "both" } {
> Yao> +	    # GDB_PERFTEST_MODE=compile|run|both is allowed.
> Yao> +	    unsupported "Unknown value of GDB_PERFTEST_MODE."
> Yao> +	    return 1
>
> This is basically an internal error.
> Calling "error" and having a backtrace in the log seems better to me.

Fixed it in the commit.  Thanks for the review.

-- 
Yao (齐尧)

Re: [PATCH 0/4 V4] GDB Performance testing

[Yao Qi]

On 11/01/2013 03:34 PM, Yao Qi wrote:
> The changes in V4 can be found in each patch.
>
> The basic usages of performance testing are,
>
> $ make check-perf
> $ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver solib.exp"
> $ make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=512"
>
> We can skip compilation step like,
>
> $ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFTEST_MODE=run'
>
> or only compile test cases,
>
> $ make check-perf RUNTESTFLAGS='solib.exp GDB_PERFTEST_MODE=compile'
>
> in default, GDB_PERFTEST_MODE=both.
>
> If the perf test case takes much time, we can specify timeout by
> GDB_PERFTEST_TIMEOUT=1000.
>
> $ make check-perf RUNTESTFLAGS="--target_board=native-gdbserver GDB_PERFTEST_TIMEOUT=4000 solib.exp"

This patch series is committed!  Thanks to Doug and Tom's patient
review.  I'll submit other test cases and try to run them nightly to 
monitor performance changes.

-- 
Yao (齐尧)

Re: [PATCH 3/4] Mention perf test in testsuite/README

[Doug Evans]

On Wed, Oct 16, 2013 at 12:09 AM, Yao Qi <yao@codesourcery.com> wrote:
> gdb/testsuite:
>
> 2013-10-16  Yao Qi  <yao@codesourcery.com>
>
>         * README: Mention performance tests.

Ok by me.

[PATCH 3/4] Mention perf test in testsuite/README

[Yao Qi]

gdb/testsuite:

2013-10-16  Yao Qi  <yao@codesourcery.com>

	* README: Mention performance tests.
---
 gdb/testsuite/README |   27 +++++++++++++++++++++++++++
 1 files changed, 27 insertions(+), 0 deletions(-)

diff --git a/gdb/testsuite/README b/gdb/testsuite/README
index ec91b14..bc06ae9 100644
--- a/gdb/testsuite/README
+++ b/gdb/testsuite/README
@@ -25,6 +25,29 @@ The second is to cd to the testsuite directory and invoke the DejaGnu
 (The `site.exp' file contains a handful of useful variables like host
 and target triplets, and pathnames.)
 
+Running the Performance Tests
+*****************************
+
+GDB Testsuite includes performance test cases, which are not run together
+with other test cases, because performance test cases are slow and need
+a quiet system.  There are two ways to run the performance test cases.
+The first is to do `make check-perf' in the main build directory:
+
+	make check-perf RUNTESTFLAGS="solib.exp SOLIB_COUNT=8"
+
+The second is to cd to the testsuite directory and invoke the DejaGnu
+`runtest' command directly.
+
+	cd testsuite
+	make site.exp
+	runtest GDB_PERFORMANCE=both GDB_PERFORMANCE_TIMEOUT=4000 --directory=gdb.perf solib.exp SOLIB_COUNT=8
+
+Only "compile", "run" and "both" are valid to GDB_PERFORMANCE.  They
+stand for "compile tests only", "run tests only", and "compile and run
+tests" respectively.  "both" is the default.  GDB_PERFORMANCE_TIMEOUT
+specify the timeout, which is 3000 in default.  The result of
+performance test is appended in `testsuite/perftest.log'.
+
 Testsuite Parameters
 ********************
 
@@ -315,6 +338,10 @@ Tests that exercise a specific GDB subsystem in more depth.  For
 instance, gdb.disasm exercises various disassemblers, while
 gdb.stabs tests pathways through the stabs symbol reader.
 
+gdb.perf
+
+GDB performance tests.
+
 Writing Tests
 *************
 
-- 
1.7.7.6