how can i make UI automation in java by which i can capture button or menu of any external application dynamically

I want to automate an external application, but I have several problems:

• How can I recognize a button or other field of an external application in Java?

• I use the Robot class in Java for making notepad automation where I open notepad, select file menu, and save or exit, etc.

• The problem is, it needs X,Y coordinates for the mouse pointer to go to the proper location.

• I want to make it more dynamic, 开发者_运维百科i.e. it should recognize the file menu of a running notepad anywhere on the desktop.

How can this be done in Java? Is there any class in Java I can use to do this?

Thanks everyone to give me response, I want to be more specific i want to know how can i make ui automation by using any tool if it is not possible in java or using any api of java.automation tool must be freeware.....i am searching net for that i found AutoIt is like that.But if any one do this type of things please share his/her experiance means is it possible to do that in AutoIt or not possible if not then which tool do that kind of things.

It is easy to integrate Sikuli into a Java-application since it is written in Java. Sikuli uses image recognition to find elements visible on the screen like buttons and such. It is very easy to use and provides an alternative for tasks that are difficult to handle with static positioning, like finding moving windows and such.

Take a look at this: http://sikuli.org/docx/faq/030-java-dev.html

Hope this helps!

You should have a look at Sikuli. It takes as inputs images of the ui elements to select an area in the targeted app. It's a UI Automation Application

That's a bit difficult to install (at least on Debian/Ubuntu, where I tested it), as you'll need a recent version of OpenCV, a particular version of JXGrabKey but the quality of the program worth the trip. Good Luck

Java doesn't have an API to examine the UI of another application; that would be a very big security risk.

Which is why the Robot class can only record events (key presses, mouse movements and clicks) but not which UI element was involved in most cases.

It would be possible to do more if the external application was written in Java because then, you could analyze the objects in memory but for obvious reasons, this isn't possible for C++ or .NET applications.