What Business Goals Are We Trying to Achieve Through Automated Testing?

Before embarking on an automation project, it will be important to articulate the problems you are trying to solve and agree with your key stakeholders on the business goals you aim to achieve using automation. 

At Coria, some common objectives we’ve set out to meet with our clients and partners include:

• Conducting real-time regression testing with every commit in certain Development, Test, or Production environments;
• Supporting efforts during a one-time data conversion or system migration;
• Ensuring data stays in-sync across disparate systems;
• Relieving some of the burden associated with repetitive, manual testing tasks;
• Providing the ability to run tests off-hours without manual intervention;
• Increasing test coverage capabilities;
• Reducing the time required to test an application;
• Reducing the cost of delivering software;
• Improving the quality of software that is delivered into Production.

How Will We Measure Success?

Here are some example metrics that matter:

• Number of bugs uncovered in Dev and Test environments using automated testing
• Number of bugs uncovered in Production that were missed before go-live during normal testing cycles

What Tools and Technologies Should We Use in the Framework?

Before you choose a technology stack, take an inventory of what skills and tools you already have on the team so you can make informed decisions when answering questions like, “Would it make more sense to write this in Java or .NET?”

What Pieces of the Framework Will Be Reusable Across Multiple Tests?

In general, good candidates for automation should be reusable (unless you’re doing a one-time validation of something like a data migration or conversion).

Will the Automated Tests Need to Run in a Virtualized Environment?

While many automated tests can churn away happily in the background while you work on something else on the same machine, at least some of your automated tests will likely require complete access to do things like open browsers and perform tasks that require the application be active on your desktop. When an error is uncovered using the tests, a screenshot of the current state of the desktop might also be taken and saved to a shared directory, so it will be important to keep this in mind when you’re multi-tasking on your laptop while it launches your test cases.

When or How Will the Automated Tests Run?

Redeployment of a website should be followed by full regression tests to determine what, if anything, may have been broken during the coding changes or redeployment. Typically, these tests can be automated to run immediately following the deployment cycle so as to relieve the IT staff from having to perform additional tasks. It may also be recommended to run tests periodically to ensure no other variables have affected the integrity of the site.

What Actions Should the Framework Take When Issues Are Uncovered?

Example actions include:

• Save a screenshot of what was visible on the screen when the error occurred;
• Send an email to a distribution list or the person that initiated the test;
• Save the results to a log file;
• Log a ticket in JIRA, TFS, or Teamwork.

Who Should Be Notified When Automated Tests Results Are Available?

It is important to consider which members of your team will benefit from the information collected by your automated testing solution. When a set of automated tests pass or fail, you most likely don’t want to blast results out to a large distribution list, because people that are not interested in the play-by-play details will learn to ignore the emails that come in and discount the value of messages that come out of the automated testing framework.

To avoid this scenario, make sure you brainstorm the flow of information to your stakeholders as you develop your automated testing solution.

What Skills and Training Will Our Team Need in Order to Run and/or Maintain Test Cases?

Whether you’re building your automated testing solution in-house or contracting with an outside firm to develop the framework for you, it is important to have a plan in place for eventually taking over the day-to-day maintenance of your automated testing.

How Will the Test Cases Stay Up-to-Date?

Ideally, each test case will start as a detailed test script written by a QA person/tester. After Automation Engineer develops these test cases and adds them to the battery of tests that are run on a regular basis, they should not disappear into the ether, never to be modified again.

We’ve worked in multiple client environments where automated test results end up being ignored because many of the failures are expected as a result of “old” test cases that haven’t been updated since the underlying functionality changed. This dilutes the value of your automated testing solution and should be avoided.

Come up with a business process around how often your test cases will be evaluated and how you will proactively incorporate updates to existing test cases as you update functionality within your application or website.

When Should Old Test Cases Be Retired?

In addition to coming up with a strategy for updating existing test cases so they do not get stale as functionality in your application changes, it is also important to make sure test cases that no longer provide value are taken out of the rotation.

Which Types of Tests Should Not Be Automated?

Though automated testing is a great way to save time and reduce the amount of tedious work required for the Quality Assurance team, there are some tests that are not good candidates for automation. Whether it’s testing the weirdness of CSS layouts in an old version of Internet Explorer or verifying that the color scheme looks right across brands, there are some tests that still require a human touch.

You should brainstorm the kinds of tests that you’d like to reserve for your human testers and do a cost-benefit analysis to ensure it makes sense to spend development time and energy automating the highest impact test cases.