iPhone test automation - benchmark tools?

Hi we are expanding one of our projects in a major bank to include access via mobile devices. We are evaluating a few tools - inc. perfecto mobile, experitest and deviceanywhere. From our initial evaluation perfecto and device anywhere cover a larger set of handsets inc feature phones. Experitest on the other hand is strong and simple to operate with smartphones(iphone, android etc). Can anyone share experience from using these tools for large scale projects? we are mainly concerened re stability, ab开发者_运维知识库ility to work with QTP and support considerations (support for new devices etc).

I have used DeviceAnywhere extensively. Perfecto, not that much, after a pretty dissapointing trial period. DA has support/add-ins for QTP and QC. Perfecto does not cover QC. Perfecto is not faster than DA, since most of their devices are in Israel, and not the US. DA has a few datacenters in the US and abroad, hence you have a better chance to get better performance. DA has an pretty long list of Enterprise and Carrier customers...while Perfecto seems like a very small company. Compare their website quality-it's pretty obvious which one looks more professional...You should try them both and make up your mind...

I have used all 3 platforms many times

Only Perfecto Mobile and DA are robust enough for real testers (at least for enterprise level).

DA have more devices but Perfecto are 100% web based, faster and MUCH cheaper. Both offer automation environments with pros and cons but Perfecto offers QTP integration and enhanced security solutions

Conclusion - both systems good, Perfecto cheaper, Perfecto much better for enterprises engaging in mobile testing.

Guido

Think of coupling a standard software remote control product with a standard software test robot (like QTP).

As an alternative, and being a mostly device-independent, but bitmap-dependent solution, you could use one of the many remote controls to bring the mobile's display contents to the desktop. Then, you'd "click around" in that remote control window using you favourite test robot.

Stupid that sounds? Well, it has its strong and its weak points:

If QTP is set for you, you'd be stuck on bitmap synchronization, no other useful GUI properties would be visible. However, if you have some QTP know-how on board, you could reuse all the know-how for test management integration via QC, test data addressing, and so on, scripting "art" like wait-for-the-right-thing, convert bitmaps to text, and so on. You could even "in real time" verify the results displayed on the mobile to stuff in the corporate backend, or research expected results in some central database after doing some transaction on the mobile -- all that would be easy since your test robot runs as part of the IT infrastructure all the time, so it has easy access to those resources. And those accesses could be done with all the comfort we got used to on PC-based test robots, like for example QTP's database checkpoint.

The positive aspect would be: Using such a scenario, you are largely independent of the mobile's technical details, and could support a lot of different devices by just using different sets of expected bitmaps. (Provided the workflows are exactly the same, which of course is not always the case.)

If you don't have to buy an extra test robot, this solution might be unbeatable cheap. Most Windows mobile devices for example can be used with Microsoft's free remote control, and there are lots of commercial vendors offering remote control functionality for a variety of devices in one package.

Also, you could develop test scripts using emulators emulating the mobile device, because the test robot would not know the difference between a display being fetched from the real thing, or being shown by the emulator.

I've done all that with various remote controls and PDA/smartphone devices, using CitraTest or QTP as the test robot. I was very happy not having to mess around with yet-another-specialized tool, or even more than one of them, each with their own language, or methodology.

Biggest hurdles besides the ones already mentioned were:

• find a remote control that is versatile, fast and reliable
• find a way to let the mobile use its "normal" communication path (for example, cellular connection) for all applications while, for performance reasons (and to minimize side effects), the remote control is connected through a direct connection (USB, propretiary synch cable, network...whatever the mobile supports).
• create a scripting "standard" which is sufficiently exact to keep test robot and mobile app execution synchronized while avoiding re-capturing expected bitmap for all supported devices too often (this can be partly automated)
• timing problems -- when you are on the bitmap level, it is hard to tell if you waited "long enough" for some message to appear, disappear, or whatever.
• cover exotics like "app continues only after you took a photo with the mobile camera". Generally speaking: Control the built-in periphery (what a contradiction...) of the mobile (in my case, I had to make the barcode scanner "see" specific images -- quite difficult and usually very device-dependent to automate)

It's feasable, though, and such a solution can be very stable and realiable, with a sufficient grade of cost-efficiency in terms of test maintenance effort (depending on what changes how frequently in the app-to-test, of course).

jQuery runs a lot of tests automatically on both feature phones and smart phones, maybe you can use their test system. As a side note, check if jQuery mobile is for you, it seems very cool.

To the best of my knowledge Perfecto Mobile has made some major improvments to its offering and currently offers some major benefits over the others, including price. In the last few months they've added popular devices like Lenovo nePaone. You can see the full list om their website:www.perfectomobile.com. Since they use differentontrol technology than Device Anywhere they are capabable of supporting new devices really quickly. Regarding stability and QTP they also have many advantages over the others. For instance, tools to record your own specific user scenarios and test them repeatedly across devices - this is a great automation tool for large scale projects.

If you are testing bank application you should consider the security issue. How do you protect your application and application data. Once you release a phone, someone else can get control over it. My recommendation is to use the on-site capabilities I believe all the above solutions have.