Automatically positioned mouse movements are off target

Automatically positioned mouse movements are off target

[Brian Sheppard]

I installed Cygwin/X (latest version, according to the Cygwin installer) on
my laptop. I use Putty with X11 forwarding to connect to a Red Hat Linux
system. I start the X windows server on my laptop using startxwin.bat as
installed. After logging in to the Red Hat system, I execute gnome-session&,
and the gnome desktop shows on my Windows desktop. So far, so good.

I am testing a Java application using a tool called Abbot. Abbot launches
your Java Swing app within Abbot’s JVM. Abbot reads the coordinates of Swing
components from internal Java objects, and then issues mouse and keystroke
commands to simulate user actions according to a script.

My observation is that the mouse clicks are off target. Specifically, Abbot
is aiming to hit the exact center of each component, but it misses either
high or low. (Left and right centering seem fine.)

The amount of the miss has something to do with my Task Bar. When the
Windows Task Bar is locked at top, then the clicks miss below (i.e., lower
on the screen) the intended component. When the Task Bar is docked to the
right, left, or bottom, or if it is at the top and set to auto-hide then
mouse clicks miss above the intended component.

I have verified that this issue is not related to Java Look & Feel: I tested
the Metal, Motif, and GTK options, and all Looks & Feels behaved the same.

I tested starting XWin with –rootless, -nodecoration, -swcursor, and
–fullscreen. These did not seem to make a difference.

Thinking that the issue might be in Abbot, I posted the question to the
Abbot-users mailing list. The reply from Abbot’s principal developer is
appended.

The developer gave me a task (to use java.awt.Robot to click at known
positions), which I have not carried out as such. It seemed like there was a
simpler way to isolate the problem.

I verified that manual mouse clicks activate the right controls. That is, if
I place the mouse over a pixel right on the border of a component, and then
click, then the click does stimulate the right component. This lends some
support to his theory that the problem occurs in automating the positioning
the mouse, since manual positioning works fine.

While carrying out this test, I noticed another anomaly. There is a text box
on the screen, and the mouse cursor should change to the “Text” cursor (the
I-beam) when hovering over that component. The cursor does change to an
I-beam, but it occurs when hovering over a rectangle that is above the
component.

Thanks,
Brian


>> Is Abbot incorrectly calculating the position of components in this 
>> configuration?

>Abbot uses AWT/Swing methods to determine the absolute position of a
component on the screen.

>It's more likely that Java running under X11 is getting incorrect
information from the X Server on your Windows box, which is in turn
confusing the Abbot operations.  It's also possible that the X Server's
XTEST extension (which is what java.awt.Robot uses to generate events) has
bugs.

>Position a window at several known locations and use java.awt.Robot to
click at absolute positions.  If the resulting events occur exactly where
you sent them, then the bug is in the abbot library.  If not, the bug is in
the X Server XTEST implementation or in the information being provided the X
Client (java in this case) by your Windows X server.



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://x.cygwin.com/docs/
FAQ:                   http://x.cygwin.com/docs/faq/

RE: Automatically positioned mouse movements are off target

[Mike Ayers]

> From: cygwin-xfree-owner@cygwin.com [mailto:cygwin-xfree-
> owner@cygwin.com] On Behalf Of Brian Sheppard

> I am testing a Java application using a tool called Abbot. Abbot
> launches
> your Java Swing app within Abbot’s JVM. Abbot reads the coordinates of
> Swing
> components from internal Java objects, and then issues mouse and
> keystroke
> commands to simulate user actions according to a script.

	On what machine is the application running?


	Thanks,

Mike

Re: Automatically positioned mouse movements are off target

[Jon TURNEY]

On 24/08/2009 23:05, Brian Sheppard wrote:
> I installed Cygwin/X (latest version, according to the Cygwin installer) on
> my laptop. I use Putty with X11 forwarding to connect to a Red Hat Linux
> system. I start the X windows server on my laptop using startxwin.bat as
> installed. After logging in to the Red Hat system, I execute gnome-session&,
> and the gnome desktop shows on my Windows desktop. So far, so good.
>
> I am testing a Java application using a tool called Abbot. Abbot launches
> your Java Swing app within AbbotÂ’s JVM. Abbot reads the coordinates of Swing
> components from internal Java objects, and then issues mouse and keystroke
> commands to simulate user actions according to a script.
>
> My observation is that the mouse clicks are off target. Specifically, Abbot
> is aiming to hit the exact center of each component, but it misses either
> high or low. (Left and right centering seem fine.)
>
> The amount of the miss has something to do with my Task Bar. When the
> Windows Task Bar is locked at top, then the clicks miss below (i.e., lower
> on the screen) the intended component. When the Task Bar is docked to the
> right, left, or bottom, or if it is at the top and set to auto-hide then
> mouse clicks miss above the intended component.

Thanks for the problem report.

I did some testing against Xserver 1.7.1 with xevent [1] and xdotool [2] to 
position the mouse pointer, but I wasn't able to reproduce the behaviour you 
are seeing.

I really have no idea how to get started using abbot to reproduce the problem 
you see, a simple test case would help a lot here.

[1] http://www.isv.uu.se/~ziemann/xevent/
[2] http://www.semicomplete.com/projects/xdotool/

-- 
Jon TURNEY
Volunteer Cygwin/X X Server maintainer

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://x.cygwin.com/docs/
FAQ:                   http://x.cygwin.com/docs/faq/

RE: Automatically positioned mouse movements are off target

[Brian Sheppard]

I addressed my problem using a different X server (xming). There was also a
commercial product that worked; I cannot remember which one.

For Xming, the key was to start it up in a specific windowing mode.
Something like full screen with no title bar.

I think that I tried all combinations of options on Cygwin-X and they all
failed, but maybe I did something wrong.

For me, the issue is resolved. But if you want to pursue it then Abbot
includes a test case that you can use to reproduce it. The Abbot installer
includes an app and a test script, and instructions
(http://abbot.sourceforge.net/doc/quick-start.shtml). You would install
Abbot on the Unix box and use X to view the execution of the test script.
When the script tries to click on a button, it will miss.

Brian

-----Original Message-----
From: Jon TURNEY [mailto:jon.turney@dronecode.org.uk] 
Sent: Friday, November 13, 2009 10:30 AM
To: cygwin-xfree@cygwin.com
Cc: sheppardco@aol.com
Subject: Re: Automatically positioned mouse movements are off target

On 24/08/2009 23:05, Brian Sheppard wrote:
> I installed Cygwin/X (latest version, according to the Cygwin 
> installer) on my laptop. I use Putty with X11 forwarding to connect to 
> a Red Hat Linux system. I start the X windows server on my laptop 
> using startxwin.bat as installed. After logging in to the Red Hat 
> system, I execute gnome-session&, and the gnome desktop shows on my
Windows desktop. So far, so good.
>
> I am testing a Java application using a tool called Abbot. Abbot 
> launches your Java Swing app within Abbot's JVM. Abbot reads the 
> coordinates of Swing components from internal Java objects, and then 
> issues mouse and keystroke commands to simulate user actions according to
a script.
>
> My observation is that the mouse clicks are off target. Specifically, 
> Abbot is aiming to hit the exact center of each component, but it 
> misses either high or low. (Left and right centering seem fine.)
>
> The amount of the miss has something to do with my Task Bar. When the 
> Windows Task Bar is locked at top, then the clicks miss below (i.e., 
> lower on the screen) the intended component. When the Task Bar is 
> docked to the right, left, or bottom, or if it is at the top and set 
> to auto-hide then mouse clicks miss above the intended component.

Thanks for the problem report.

I did some testing against Xserver 1.7.1 with xevent [1] and xdotool [2] to
position the mouse pointer, but I wasn't able to reproduce the behaviour you
are seeing.

I really have no idea how to get started using abbot to reproduce the
problem you see, a simple test case would help a lot here.

[1] http://www.isv.uu.se/~ziemann/xevent/
[2] http://www.semicomplete.com/projects/xdotool/

--
Jon TURNEY
Volunteer Cygwin/X X Server maintainer


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://x.cygwin.com/docs/
FAQ:                   http://x.cygwin.com/docs/faq/