BruteForce Lab's Blog

I recently completed a project with the goal of automating our functional testing infrastructure. I should actually call it semi-automation since a team of human engineers are still needed in order to build and fire test runs (which is not a downside, it was designed with that in mind). Below I will describe a rough plan for everybody who wants to do something similar. I won’t go into much details but I will surely give an overall workflow that is easy to follow.

The challenge here was mostly the integration of TestRail with the local QA environment, and this is what I am going to describe. If you don’t use TestRail, I am not sure if this will be of any benefit to you.

The tools

I suppose that to search for this article you have an idea of how to implement testing automation and have already installed some of the needed tools. In any case here are some of the tools I used:

1. TestRail: TestRail is described as a “Test Case Management & Test Management Software Tool”, which is actually a pretty good description. You can think of it like a CMS to organize your tests. You can create different projects, categories, test suites and test cases. And then you can take some of these test suites as a whole or specific test cases and group them in what is called a “test run”. A test run is let’s say a runnable instance of the selected tests. The goal of the overall project was to automate the actual execution of each test run and report back the results to TestRail.
2. Amazon Web Services (AWS): One or more servers where our tests would run on were needed, so some commodity EC2 instances served that need. You don’t have to buy something powerful to begin, a small EC2 instance can be enough.
3. Selenium: In particular, Selenium Server and WebDriver, in order to run each test properly since they actually automate browser functionality and actions. Using a combination of those on the same machine you can either run the tests locally or remotely (perhaps on a cloud service like Sauce Labs or BrowserStack).
4. Python with unittest to write the actual functional tests using the WebDriver API and py.test to execute the tests (perhaps in parallel) and get back their results.
5. Headless Firefox, Chrome, Opera, PhantomJS and Xvfb to emulate a screen.
6. Other standard languages/technologies like PHP, MySQL, git, cron, etc.

Integration workflow

Bear in mind that this is not what was eventually implemented (as it gets more complex down the road with multiple servers, parallel testing, etc) but it was my first working prototype and it should serve as a good basis to start with.

So here is how the TestRail integration works:

Read the rest of this entry »