Using Robot Framework for ATDD

I would like to hear other people's experience with Robot Framework for automated acceptance testing.

What are its major strengths and weaknesses as well as any comparison with other frameworks (mainly Fitnes开发者_开发百科se and Selenium)?

The code that will be tested is real-time, legacy code, mainly in C++.

I have used Robot Framework at three different companies spanning about six years at the time that I write this, and all have been successful in one way or another.

My experiences

Company 1

The first place I used Robot was a Java-based web application for a top-tier Internet travel company. We used Robot with Jython, which let us create keywords in Java and directly act with the system under test. We used Selenium to drive the web browser, with most of our testing being on Firefox. While the testing effort was largely successful with the QA organization, the development organization failed to embrace it -- they preferred to use JUnit rather than Robot.

Company 2

My second company I feel was an unqualified success. We used Robot in a multitude of ways. The primary use was to drive Internet Explorer for acceptance and regression testing of a highly successful commercial .NET web application.

We also used it to test an iPad app by combining Selenium with Appium. We used Robot to test the RESTful services that supplied data to the app. We wrote specialized keywords that let us do image analysis, and we also used Robot tests to do quick analysis of our training equipment before each training session. We had keywords that let us snapshot a database before a test, and to restore the database after a test.

We also started using Robot to help with manual testing. We put manual test cases in Robot, which let us leverage Robot's reporting and tagging features. When these tests would run, they would prompt the user to perform manual steps which was proven to be much more efficient than when we had testers reading manual steps out of a test case management tool or Word document.

Company 3

The third company was a large company ($1B revenue) with a fairly large IT staff. They had testers with very low technical skill (I remember one who had no idea what a command line was). We had one team dedicated to writing a core set of keywords, and to providing training and mentoring to the other teams. I think the use of Robot was instrumental in getting some use out of the least-skilled testers, though even with high level keywords it was a struggle with them.

Company 4

Most recently I moved to a very small company with just a handful of developers and no dedicated testers. We embraced the use of page objects with Robot Framework, and we now have an exceptionally stable suite of easily readable high level acceptance tests. The use of Robot at this company has been an unqualified success.

Strengths

The biggest strength of Robot is its flexibility. We've used Robot to support manual testing, testing of SOAP and REST services, web-based UI testing, database testing, testing of images, and the testing of mobile apps. Because Robot is so easy to extend with additional libraries, there's almost nothing you can't test if you're willing to roll up your sleeves and write some keywords. Depending on your setup, you can write keywords in Python, Java, .NET, or really just about any language through the Robot remote API.

Because Robot test cases and keywords are written in plain text, you're not locked in to using a proprietary tool to create or view tests. Users are able to pick the tool of their choice -- Visual Studio, Eclipse, Emacs, Notepad, etc. There is also a Robot-specific IDE (RIDE), though I don't recommend it. Also, because the files are plain text, they integrate well with other software tools -- they are easy to diff and merge, to search, etc.

Weaknesses

Robot makes it easy to write low-quality tests. While there are facilities to document keywords and test cases, and to use human-readable names for keywords, test cases and variables, there's no good way to enforce best practices. Writing a large body of tests and keywords requires discipline. As the saying goes, Robot gives you plenty of rope to hang yourself with.

Another weakness is that the pace of progress on Robot is fairly slow. On the plus side, Robot is robust and relatively bug free, so there's not a big need for frequent patches. However, there are feature requests that languish in their issue tracker for years with no movement, which can be discouraging.

Summary

In all companies we've enjoyed being able to leverage the flexibility provided by Robot's syntax to create data-driven tests, BDD-style tests, as well as simple procedural tests. And in all cases, because the tests are plain text files, the test assets were easy to manage with our SCM tools (Mercurial, Subversion, and Git)

For me, Robot has proven to be easy to use, extremely easy to extend, and useful for a wide range of testing duties, from unit-testing of Python functions, to testing of web services, browser-based and tablet UI testing, the testing of images, the testing of databases, and even to improve the efficiency of manual testing.

We have been using Robot Framework at my place of work for several over a year now with moderate success. Like the poster, we also do C++ work. We took some time to evaluate Robot against Fitnesse/Slim and, at the time, both solutions were good, but the deciding factors were (as of ~2009):

• It was clearer how Robot and its reports would scale to large projects
• It wasn't obvious how to version control Fitnesse artifacts

From a technical perspective, we have been using SWIG to bridge between Robot and C++. We wrap our test fixtures in SWIG and link it with the production code under test - giving us a python module that can be imported by Robot.

We use the .txt format for Robot input almost exclusively - we've found that this version controls better, it's easier to read, and we simply weren't using the advanced features of HTML (which is where we started). In addition, we are using the "BDD Style" Robot syntax as well. We use GoogleMock with some wrappers to help us set expectations which are checked during the teardown of each Robot test.

As far as organizing the tests, we have settled on the following approach (with inspiration from Dale Emery's approach given here):

• Major functional hierarchy is represented by a folder structure.
• A feature-ish sized thing is described in a Robot test file name.
• A description of each part of that feature is used at the Robot test case name.
• An example is given as the steps in the test case.
• The example text is broken down into steps using Robot "keywords".
• The test fixture drives the production code.

For example, a phone might have something like this:

//  PhoneProject/MakingCalls/DialAPhoneNumber.txt

*** Test Case ***
A user can dial a US number with an area code, up to 10 digits
   Given a phone without any numbers dialed
   Expect the attempted phone number to be 123-456-7890
   When a user enters the numbers 1234567890


// A separate MakingCallsKeywords.txt or something similar
*** Keyword ***
Given a phone without any numbers dialed                CreateNewDialer
Expect the attempted phone number to be ${phoneNumber}  ExpectDialedNumber  ${phoneNumber}
When a user enters the numbers ${numbersEntered}        DialNumbers ${numbersEntered}

// MakingCallsFixture.cpp  (which gets wrapped by SWIG)

std::wstring MakingCallsFixture::DialNumbers(const std::wstring& numbersEntered)
{
   ... Drive production code ...
}

// CreateNewDialer, ExpectDialedNumber also go here

We would then pair this up with unit tests that cover the corner conditions (e.g. ensure no more than 10 digits are allowed) - or maybe that would be another acceptance test (executable spec) depending on who is reading the tests and their familiarity with the domain.

We create a draft of these specs and review with domain experts and the team prior to beginning work. This has helped flush out a number of misunderstandings early on.

I used Robot framework for following scenarios.

UI testing

1. The Robot framework is really good for testing user interface
2. The framework comes with the RIDE editor which can be used for writing test cases.
3. You do not have to be an expert on Selenium and Selenium commands are searchable in the RIDE editor.
4. You can have shared code by using keywords and shared resources. It allows you to simply reuse test cases and test library code.
5. You can call easily call a Python library function to do more tedious tasks.

For service testing, I found the Robot framework a little difficult to use for the kind of test automation I was undertaking.

Our application service layer is completely written in C/C++ . I personally work on a Windows laptop and our application resides on a Linux server.

I found the *Fitnesse framework** much more helpful and simpler to do automation with. Fitnesse had the below mentioned extra features which allowed to write test cases easily. I could not find similar features in Robot.

Decision table - You can write test cases in Microsoft .xls kind of format. Each row in the data grid will represent a test case. Each row will have a set of inputs and outputs. Outputs will be preceded by question mark in the header.

Query Table - Output of test will be list of data which you would want to validate.

Also Fitnesse allows easy integration with other languages like C (using Slim Service). This does not require any integration coding. Fitnesse test cases directly execute test fixtures, getters, and setters.

Summary of my experience:

• I found an easy-to-use tool for UI automation.

• I found it a little difficult to use for service automation.