Patch: GUI testing framework

Patch: GUI testing framework

[Thomas Fitzsimmons]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

Hi,

This patch adds support to batch_run for running GUI tests, either
within an Xvfb server (default) or directly on the tester's desktop.  It
also introduces the GUI tag; all tests that require DISPLAY to be set
should be tagged as GUI tests.

I tested this with the java.awt.Robot tests I posted last week.  The
tests run properly both within Xvfb and on my desktop.

I would like to rely on the metacity window manager, since it handles
inset sizing correctly.

Comments?
Tom

2005-01-03  Thomas Fitzsimmons  <fitzsim@redhat.com>

	* batch_run: Invoke Xvfb and metacity if GUI key is specified.
	* choose: Handle GUI tag.
	* README: Add section on GUI tests.


[-- Attachment #2: Type: text/x-patch, Size: 2859 bytes --]

Index: README
===================================================================
RCS file: /cvs/mauve/mauve/README,v
retrieving revision 1.18
diff -u -r1.18 README
--- README	12 Aug 2004 15:11:12 -0000	1.18
+++ README	3 Jan 2005 23:23:53 -0000
@@ -172,6 +172,15 @@
 
 ================================================================
 
+Graphical tests that display windows and accept input are marked with
+the GUI tag.  GUI tests are not included in the default list of tags.
+If the GUI tag does appear in KEYS, batch_run will spawn an Xvfb
+process, set DISPLAY to that X server and run the
+graphical/interactive tests there.  If you'd rather run the tests
+directly on your desktop, set SHOW_GUI_TESTS=1.
+
+================================================================
+
 The test harness can also ignore known failures.  Simply create a
 file 'xfails' in the directory where the tests are being run which
 contains 'FAIL:' entries from previous test runs.  The order of the
Index: batch_run
===================================================================
RCS file: /cvs/mauve/mauve/batch_run,v
retrieving revision 1.5
diff -u -r1.5 batch_run
--- batch_run	19 Nov 2004 10:09:28 -0000	1.5
+++ batch_run	3 Jan 2005 23:23:53 -0000
@@ -63,6 +63,30 @@
 # Generate test classes list
 ./choose-classes . "$KEYS"
 
+case $KEYS in
+    *GUI*)
+
+        XVFB=`which Xvfb`
+        METACITY=`which metacity`
+
+        if test -z "$SHOW_GUI_TESTS"; then
+            if test -z $XVFB; then
+		echo "warning: Xvfb not found on PATH.  Showing GUI tests on main desktop..."
+	    elif test -z $METACITY; then
+		echo "warning: metacity not found on PATH.  Showing GUI tests on main desktop..."
+            else
+		export DISPLAY=:57
+		echo localhost > Xvfb.cfg
+		$XVFB :57 -auth Xvfb.cfg > /dev/null 2>&1 &
+		XVFB_PID=$!
+		$METACITY > /dev/null 2>&1 &
+		METACITY_PID=$!
+		rm -f Xvfb.cfg
+	    fi
+	fi
+	;;
+esac
+
 # Compile and run all the tests
 for testclass in `cat test_classes`; do
     # Get all needed java sources
@@ -81,3 +105,13 @@
         echo "FAIL: $testclass COMPILE FAILED"
     fi
 done
+
+case $KEYS in
+    *GUI*)
+
+        # killing Xvfb also kills the window manager process
+        if test ! -z "$XVFB_PID"; then
+            kill $XVFB_PID > /dev/null 2>&1
+        fi
+	;;
+esac
Index: choose
===================================================================
RCS file: /cvs/mauve/mauve/choose,v
retrieving revision 1.25
diff -u -r1.25 choose
--- choose	14 Oct 2004 06:53:07 -0000	1.25
+++ choose	3 Jan 2005 23:23:53 -0000
@@ -69,6 +69,10 @@
        # You can't specify this.
        :
        ;;
+    GUI)
+       tags="$tags GUI $i"
+       test -z "$seen_dashdash" && spectags="$spectags $i"
+       ;;
     JAVA2)
        tags="$tags JDK1.0 JDK1.1 JDK1.2 $i"
        test -z "$seen_dashdash" && spectags="$spectags $i"

Re: Patch: GUI testing framework

[Thomas Zander]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

On Tuesday 04 January 2005 00:36, Thomas Fitzsimmons wrote:
> I would like to rely on the metacity window manager, since it handles
> inset sizing correctly.

If you feel the KWin WM does not, please make sure to inform the KWin 
developers where they don't follow the spec.
I'd find it slightly amuzing if KDE does not since at least the main 
developer is also part of the spec lead..

If you don't want to create a bugreport, please tell me exactly what the 
problem is (a reproducable c/c++ testcase would be great) so I can 
contact Lubos about this.

-- 
Thomas Zander

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Patch: GUI testing framework

[Michael Koch]

Am Dienstag, 4. Januar 2005 00:36 schrieb Thomas Fitzsimmons:
> Hi,
>
> This patch adds support to batch_run for running GUI tests, either
> within an Xvfb server (default) or directly on the tester's
> desktop.  It also introduces the GUI tag; all tests that require
> DISPLAY to be set should be tagged as GUI tests.
>
> I tested this with the java.awt.Robot tests I posted last week. 
> The tests run properly both within Xvfb and on my desktop.

Nice. Classpath has it now too, so we can generally test it too 
directly.

> I would like to rely on the metacity window manager, since it
> handles inset sizing correctly.

Depending on a specific window manager is stupid. It just makes the 
hurdles for users of mauve higher. If we have problems with a 
specific window manager and its his fault and not ours we should 
report bugs.

If you refer to the wrong insets bug I see here, I made sure its a 
jamvm/classpath bug, not a window manager bug. It only occurs in 
jamvm/classpath and not in a 100% merged java-gui-branch. I will try 
to investigate it more, no matter if I run kwin, windowmaker or 
GNOME/metacity.


Michael
-- 
Homepage: http://www.worldforge.org/

Re: Patch: GUI testing framework

[Thomas Fitzsimmons]

On Tue, 2005-01-04 at 10:02 +0100, Michael Koch wrote:
> Am Dienstag, 4. Januar 2005 00:36 schrieb Thomas Fitzsimmons:
> > Hi,
> >
> > This patch adds support to batch_run for running GUI tests, either
> > within an Xvfb server (default) or directly on the tester's
> > desktop.  It also introduces the GUI tag; all tests that require
> > DISPLAY to be set should be tagged as GUI tests.
> >
> > I tested this with the java.awt.Robot tests I posted last week. 
> > The tests run properly both within Xvfb and on my desktop.
> 
> Nice. Classpath has it now too, so we can generally test it too 
> directly.
> 
> > I would like to rely on the metacity window manager, since it
> > handles inset sizing correctly.
> 
> Depending on a specific window manager is stupid. It just makes the 
> hurdles for users of mauve higher. If we have problems with a 
> specific window manager and its his fault and not ours we should 
> report bugs.
> 

OK.  Can we default to metacity?  I'm thinking of:

if test "x$WM" = "x" ; then
    WM=metacity
fi

Then you could set WM on the command line.

Tom

Re: Patch: GUI testing framework

[Michael Koch]

Am Dienstag, 4. Januar 2005 17:39 schrieb Thomas Fitzsimmons:
> On Tue, 2005-01-04 at 10:02 +0100, Michael Koch wrote:
> > Am Dienstag, 4. Januar 2005 00:36 schrieb Thomas Fitzsimmons:
> > > Hi,
> > >
> > > This patch adds support to batch_run for running GUI tests,
> > > either within an Xvfb server (default) or directly on the
> > > tester's desktop.  It also introduces the GUI tag; all tests
> > > that require DISPLAY to be set should be tagged as GUI tests.
> > >
> > > I tested this with the java.awt.Robot tests I posted last week.
> > > The tests run properly both within Xvfb and on my desktop.
> >
> > Nice. Classpath has it now too, so we can generally test it too
> > directly.
> >
> > > I would like to rely on the metacity window manager, since it
> > > handles inset sizing correctly.
> >
> > Depending on a specific window manager is stupid. It just makes
> > the hurdles for users of mauve higher. If we have problems with a
> > specific window manager and its his fault and not ours we should
> > report bugs.
>
> OK.  Can we default to metacity?  I'm thinking of:
>
> if test "x$WM" = "x" ; then
>     WM=metacity
> fi
>
> Then you could set WM on the command line.

Why do you need the window manager at all ? Where does the $WM value 
get used ?


Michael
-- 
Homepage: http://www.worldforge.org/

Re: Patch: GUI testing framework

[Thomas Fitzsimmons]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

On Tue, 2005-01-04 at 09:30 +0100, Thomas Zander wrote:
> On Tuesday 04 January 2005 00:36, Thomas Fitzsimmons wrote:
> > I would like to rely on the metacity window manager, since it handles
> > inset sizing correctly.
> 
> If you feel the KWin WM does not, please make sure to inform the KWin 
> developers where they don't follow the spec.
> I'd find it slightly amuzing if KDE does not since at least the main 
> developer is also part of the spec lead..
> 

It's not that they don't follow the spec, it's just that they don't
interpret it as strictly as metacity does.  See this thread for the
original discussion:

http://mail.gnome.org/archives/wm-spec-list/2003-December/msg00005.html

> If you don't want to create a bugreport, please tell me exactly what the 
> problem is (a reproducable c/c++ testcase would be great) so I can 
> contact Lubos about this.
> 

Try running gnu.testlet.java.awt.Robot.mouseMove under KWin and then
under Metacity.  I've attached a new patch that makes it easier to
override the default window manager.

Tom

2005-01-03  Thomas Fitzsimmons  <fitzsim@redhat.com>

	* batch_run: Invoke Xvfb and metacity if GUI key is specified.
	* choose: Handle GUI tag.
	* README: Add section on GUI tests.


[-- Attachment #2: Type: text/x-patch, Size: 2994 bytes --]

Index: README
===================================================================
RCS file: /cvs/mauve/mauve/README,v
retrieving revision 1.18
diff -u -r1.18 README
--- README	12 Aug 2004 15:11:12 -0000	1.18
+++ README	4 Jan 2005 18:05:58 -0000
@@ -172,6 +172,17 @@
 
 ================================================================
 
+Graphical tests that display windows and accept input are marked with
+the GUI tag.  GUI tests are not included in the default list of tags.
+If the GUI tag does appear in KEYS, batch_run will spawn an Xvfb
+process, set DISPLAY to that X server and run the
+graphical/interactive tests there.  By default, metacity is run in
+Xvfb; use the WM environment variable to run a different window
+manager.  If you'd rather run the tests directly on your desktop, set
+SHOW_GUI_TESTS=1.
+
+================================================================
+
 The test harness can also ignore known failures.  Simply create a
 file 'xfails' in the directory where the tests are being run which
 contains 'FAIL:' entries from previous test runs.  The order of the
Index: batch_run
===================================================================
RCS file: /cvs/mauve/mauve/batch_run,v
retrieving revision 1.5
diff -u -r1.5 batch_run
--- batch_run	19 Nov 2004 10:09:28 -0000	1.5
+++ batch_run	4 Jan 2005 18:05:58 -0000
@@ -63,6 +63,32 @@
 # Generate test classes list
 ./choose-classes . "$KEYS"
 
+case $KEYS in
+    *GUI*)
+
+        XVFB=`which Xvfb`
+        if test "x$WM" = "x"; then
+            WM=`which metacity`
+        fi
+
+        if test -z "$SHOW_GUI_TESTS"; then
+            if test -z $XVFB; then
+		echo "warning: Xvfb not found on PATH.  Showing GUI tests on main desktop..."
+	    elif test -z $WM; then
+		echo "warning: metacity not found on PATH.  Showing GUI tests on main desktop..."
+            else
+		export DISPLAY=:57
+		echo localhost > Xvfb.cfg
+		$XVFB :57 -auth Xvfb.cfg > /dev/null 2>&1 &
+		XVFB_PID=$!
+		$WM > /dev/null 2>&1 &
+		WM_PID=$!
+		rm -f Xvfb.cfg
+	    fi
+	fi
+	;;
+esac
+
 # Compile and run all the tests
 for testclass in `cat test_classes`; do
     # Get all needed java sources
@@ -81,3 +107,13 @@
         echo "FAIL: $testclass COMPILE FAILED"
     fi
 done
+
+case $KEYS in
+    *GUI*)
+
+        # killing Xvfb also kills the window manager process
+        if test ! -z "$XVFB_PID"; then
+            kill $XVFB_PID > /dev/null 2>&1
+        fi
+	;;
+esac
Index: choose
===================================================================
RCS file: /cvs/mauve/mauve/choose,v
retrieving revision 1.25
diff -u -r1.25 choose
--- choose	14 Oct 2004 06:53:07 -0000	1.25
+++ choose	4 Jan 2005 18:05:58 -0000
@@ -69,6 +69,10 @@
        # You can't specify this.
        :
        ;;
+    GUI)
+       tags="$tags GUI $i"
+       test -z "$seen_dashdash" && spectags="$spectags $i"
+       ;;
     JAVA2)
        tags="$tags JDK1.0 JDK1.1 JDK1.2 $i"
        test -z "$seen_dashdash" && spectags="$spectags $i"

Re: Patch: GUI testing framework

[Thomas Fitzsimmons]

On Tue, 2005-01-04 at 18:36 +0100, Michael Koch wrote:
> Am Dienstag, 4. Januar 2005 17:39 schrieb Thomas Fitzsimmons:
> > On Tue, 2005-01-04 at 10:02 +0100, Michael Koch wrote:
> > > Am Dienstag, 4. Januar 2005 00:36 schrieb Thomas Fitzsimmons:
> > > > Hi,
> > > >
> > > > This patch adds support to batch_run for running GUI tests,
> > > > either within an Xvfb server (default) or directly on the
> > > > tester's desktop.  It also introduces the GUI tag; all tests
> > > > that require DISPLAY to be set should be tagged as GUI tests.
> > > >
> > > > I tested this with the java.awt.Robot tests I posted last week.
> > > > The tests run properly both within Xvfb and on my desktop.
> > >
> > > Nice. Classpath has it now too, so we can generally test it too
> > > directly.
> > >
> > > > I would like to rely on the metacity window manager, since it
> > > > handles inset sizing correctly.
> > >
> > > Depending on a specific window manager is stupid. It just makes
> > > the hurdles for users of mauve higher. If we have problems with a
> > > specific window manager and its his fault and not ours we should
> > > report bugs.
> >
> > OK.  Can we default to metacity?  I'm thinking of:
> >
> > if test "x$WM" = "x" ; then
> >     WM=metacity
> > fi
> >
> > Then you could set WM on the command line.
> 
> Why do you need the window manager at all ? Where does the $WM value 
> get used ?
> 

You don't need a window manager, but having one makes for a more
realistic testing environment.  With the patch I just posted, you can
set WM=/bin/true to run without a window manager.

Tom

Re: Patch: GUI testing framework

[Tom Tromey]

>>>>> "Tom" == Thomas Fitzsimmons <fitzsim@redhat.com> writes:

Tom> if test "x$WM" = "x" ; then
Tom>     WM=metacity
Tom> fi

That seems reasonable to me.  The patch looks good too.

Tom

Re: Patch: GUI testing framework

[Michael Koch]

Am Dienstag, 4. Januar 2005 17:39 schrieb Thomas Fitzsimmons:
> On Tue, 2005-01-04 at 10:02 +0100, Michael Koch wrote:
> > Am Dienstag, 4. Januar 2005 00:36 schrieb Thomas Fitzsimmons:
> > > Hi,
> > >
> > > This patch adds support to batch_run for running GUI tests,
> > > either within an Xvfb server (default) or directly on the
> > > tester's desktop.  It also introduces the GUI tag; all tests
> > > that require DISPLAY to be set should be tagged as GUI tests.
> > >
> > > I tested this with the java.awt.Robot tests I posted last week.
> > > The tests run properly both within Xvfb and on my desktop.
> >
> > Nice. Classpath has it now too, so we can generally test it too
> > directly.
> >
> > > I would like to rely on the metacity window manager, since it
> > > handles inset sizing correctly.
> >
> > Depending on a specific window manager is stupid. It just makes
> > the hurdles for users of mauve higher. If we have problems with a
> > specific window manager and its his fault and not ours we should
> > report bugs.
>
> OK.  Can we default to metacity?  I'm thinking of:
>
> if test "x$WM" = "x" ; then
>     WM=metacity
> fi

Can you default to /bin/true when metacity is not in the path ?


Michael
-- 
Homepage: http://www.worldforge.org/

Re: Patch: GUI testing framework

[Thomas Fitzsimmons]

On Tue, 2005-01-04 at 13:08 -0500, Thomas Fitzsimmons wrote:

> Try running gnu.testlet.java.awt.Robot.mouseMove under KWin and then
> under Metacity.  I've attached a new patch that makes it easier to
> override the default window manager.

I committed this.

Tom