Test Case Management Software Integration

Mixing automated and manual testing

This is a two part post on mixing automated and manual testing (Part 2). In short, I cover solutions to two problems:

  • Converting selenium tests to end-to-end unit tests
  • Reporting end-to-end test to a manual test case management system

 

A few months ago, I built a Continuous Integration and Deployment platform for a client. The CI/CD system was nice with automated unit testing, code analysis, and automated deployment to a testing environment. There remained two gaps though:

  • Much testing was still manual
  • Many tests were automated end-to-end tests using Selenium

I was tasked with improving their process:

  • Convert end-to-end test to unit tests
  • Report unit test failures to their manual testing platform, TestRail

Mixing automated and manual testing is always a challenge. Most automated tests are unit tests which don’t look at the application in it’s entirety as a user. Nor do most unit test cover UI workflow. Selenium is good (meh) at automating end-to-end testing. Manual testing is a challenge because it’s time consuming and prone to laziness. It’s difficult to run all test cases all the time. The best solution is to create your Selenium tests and import them into your CI workflow. Mixing automated and manual testing becomes an exercise in technical process automation rather than a manpower issue.

Selenium Test Conversion to Unit Tests

I won’t cover how to create or export tests using the Selenium IDE. That information is covered elsewhere.

Here are a few tips for importing the exported test cases:

Create a base test class with driver defaults

var service = PhantomJSDriverService.CreateDefaultService();
service.AddArgument("--webdriver-loglevel=NONE");
driver = new PhantomJSDriver(service);
driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(5));
driver.Manage().Window.Maximize();
  1. Use the PhantomJS test driver. My Continuous Integration servers normally run headless (meaning they don’t have access to a UI thread). Using PhantomJS is a great tool for browserless browser control.
  2. Set loglevel to “NONE”. Without this, your CI console will be filled with unhelpful information
  3. Set the default timeout between steps to 3-5 seconds. Browser operations can take a bit longer on a test environment than you realize. These are all automated end-to-end tests. If you care about speed, you should be writing unit tests with dependency injection and mocking frameworks.
  4. Maximize the window. This allows all UI elements to be “in view”.

I put these in the TestInitalize method of my SeleniumBaseTest.cs. Then, I have to update all my tests cases to inherit from SeleniumBaseTest.

Import and clean up your test cases

After you import your test cases, you may notice many things you don’t like. I always do the following:

  • Delete the unused methods at the bottom
  • Change to use MSTest
  • Delete the first-step navigation
  • Remove any spaces from names

I write a little clean up script to perform these file manipulation tasks. I recommend you do the same.

Integrate into your CI

These steps vary depending on your setup. I keep my end-to-end tests in a separate project from the rest of my unit tests. These take longer to run. I don’t require developer to run them before each check-in like unit tests.

The Result

At this point you should have all your end-to-end test running at each commit and sending out failure reports. However, there are some outstanding issues you’ll likely encounter:

  • Idempotence – There’s a fun party word to impress your friends with. In short, idempotent test can run many times with out leaving side-effects. The never change the state of the application. Many end-t0-end tests depend on user accounts, subscription status, carts, shop items, or any number of things. A test run may fail in the middle leaving the database or session in an inconsistent state. You need a way to revert the changes. Good options are restoring a test database between run or truncating and reloading tables.
  • Slow tests – Active development teams which check in frequently and depend on the results of the CI server to validate their results are negatively affected by long running tests. Setting up a secondary CI server to run end-to-end is usually a good idea.

 

Wm. Barrett Simms
Follow Me
Latest posts by Wm. Barrett Simms (see all)