04:09 PM
Jeremy Greshin, Telesis
Jeremy Greshin, Telesis
Connect Directly

Simply Building a Testing Automation Framework Won't Save Your ROI

Done correctly, automated software testing consists of three parts: creating the automation architecture, writing the business test scenarios, and assessing the testing state of the software.

Automated software testing presents serious challenges to insurance companies. The more efficient the testing is, the more testing gets done and the fewer errors get into production. The less efficient the testing, the more time it takes, until the cost of automated testing exceeds the amount of money automation was supposed to save the insurance company. In that case, any return on investment disappears. Done correctly, automated software testing consists of three parts: (1) creating the automation architecture, (2) writing the business test scenarios, and (3) assessing the testing state of the software.

Step One: Create the Automation Architecture. Creating the right automation architecture is the most important step. Initially, that requires selecting and implementing the best fit functional software test tool. Literally hundreds of test tools exist on the market today, and many articles have been written about picking the right one.

Much less has been written, unfortunately, about the importance of configuring the test tool correctly. Most functional test tools are advertised as "plug and play," one size fits all. This simply is not true. In fact, most functional test tools require at least some degree of configuring and architecting for the application being tested. The better the test tool and the better it is configured to the application, the easier it is to build reusable and durable test scripts.

This is where insurance companies make a critical mistake. Instead of hiring people who know how to configure their functional test tools for their applications, insurers hire programmers to write code around the configuration problems. Programmers re-create the application code to create the test code. That is, they code twice. Coding twice means high maintenance costs because each code change to the application requires a code change to the test code. The programmers basically re-build the application. This approach has several additional downsides: (1) the scripts frequently are not reusable because of coding; (2) expensive, highly trained technical resources must build, use and maintain the code; (3) business people cannot easily utilize the automated test scripts because they do not have a technical background; (4) adequate testing becomes costly and time consuming; and (5) testing is done from the programmer's perspective, not the business's perspective.

If the functional test tool is configured correctly for the application, the test scripts are repeatable and reusable. Correct automation architecture goes well beyond just building an automation framework. Test scripts must be dependable across multiple version upgrades, without having to rework the code. Business people should be able to utilize the automation for their testing. Only then can the insurance company make sure that the software does what it is supposed to do -- by testing from the end user's perspective, not the developer's.

Step Two: Write the Business Test Scenarios. The next step is to create the test scripts to run with the automation. This requires working closely with the business people, who live with the application, to create a priority grid. The grid consists of ranking the most important to the least important business transactions in the application. For example, an auto insurance company that does 95% of its business with middle aged New England drivers purchasing $300,000 to $500,000 of insurance would want to make sure that it has tested this group thoroughly. Conversely, teenager drivers in Florida might be a low priority. The ranking system lets the user make sure that the high priority test scenarios are tested first. The user saves time and money by dedicating resources where they matter most. The rankings can be adjusted as time allows and as priorities change.

State of the art test tools can convert a single script into an array of test scripts by paramaterizing the data. For example, a health insurance company wants to test a variety of claim coverage scenarios for newborns. Instead of writing numerous scripts, the company can develop a single test script with a supporting data table, so that the script can be used to test hundreds of scenarios:

Table 1

In some cases, a single test script can be leveraged to execute over 3,000 different test scenarios. Parameterizing the data makes testing faster and easier, increases application test coverage, and reduces the number of software defects.

Step Three: Monitor the Daily Test Work. After creating the automation and writing the business scenarios, the final step is to manage the testing progress. This is where a dashboard is useful. The dashboard is a test management tool that helps the user understand the exact state of testing at any particular moment on each application under test (AUT). The dashboard serves as a report card for how well the AUT has been tested to date, and what needs to be done now. It can be used to create a test efficiency rating, for "Go/No Go" decision support, for defect impact analysis, and for project test status. Examples might include:

* System status: The percentage of critical scenarios being tested * Potential system inefficiency: The percentage of built test assets addressing low risk scenarios * Asset execution status: The percentage of critical assets executed successfully

The dashboard is an excellent tool for tying software testing directly to the business the software is in, by making sure that the most important business transactions are tested first. The dashboard should be role based so that management can quickly assess the overall AUT status and individual testers know on a daily basis what test cases need to be worked on today.

Software test automation, when done correctly, can yield anywhere from a 35% to 90% return on investment in reduced testing time, depending on the amount of existing testing that is manual. The number of errors that get into production also should decrease by over 75%. If your company is not seeing these numbers, most likely the fault lies with the way the automation is architected or how the dashboard and business scenarios are utilized. Automated software testing is only worthwhile if it is saving time and minimizing errors.

About the Author: Jeremy Greshin is vice president for business development at Hartford-based Telesis, a provider of QA and software testing solutions. He can be reached at (860) 289-4504

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: Janice, I think I've got a message from the code father!
Current Issue
Security Operations and IT Operations: Finding the Path to Collaboration
A wide gulf has emerged between SOC and NOC teams that's keeping both of them from assuring the confidentiality, integrity, and availability of IT systems. Here's how experts think it should be bridged.
Flash Poll
New Best Practices for Secure App Development
New Best Practices for Secure App Development
The transition from DevOps to SecDevOps is combining with the move toward cloud computing to create new challenges - and new opportunities - for the information security team. Download this report, to learn about the new best practices for secure application development.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
Published: 2017-05-09
NScript in mpengine in Microsoft Malware Protection Engine with Engine Version before 1.1.13704.0, as used in Windows Defender and other products, allows remote attackers to execute arbitrary code or cause a denial of service (type confusion and application crash) via crafted JavaScript code within ...

Published: 2017-05-08
unixsocket.c in lxterminal through 0.3.0 insecurely uses /tmp for a socket file, allowing a local user to cause a denial of service (preventing terminal launch), or possibly have other impact (bypassing terminal access control).

Published: 2017-05-08
A privilege escalation vulnerability in Brocade Fibre Channel SAN products running Brocade Fabric OS (FOS) releases earlier than v7.4.1d and v8.0.1b could allow an authenticated attacker to elevate the privileges of user accounts accessing the system via command line interface. With affected version...

Published: 2017-05-08
Improper checks for unusual or exceptional conditions in Brocade NetIron 05.8.00 and later releases up to and including 06.1.00, when the Management Module is continuously scanned on port 22, may allow attackers to cause a denial of service (crash and reload) of the management module.

Published: 2017-05-08
Nextcloud Server before 11.0.3 is vulnerable to an inadequate escaping leading to a XSS vulnerability in the search module. To be exploitable a user has to write or paste malicious content into the search dialogue.

Dark Reading Radio
Archived Dark Reading Radio
In past years, security researchers have discovered ways to hack cars, medical devices, automated teller machines, and many other targets. Dark Reading Executive Editor Kelly Jackson Higgins hosts researcher Samy Kamkar and Levi Gundert, vice president of threat intelligence at Recorded Future, to discuss some of 2016's most unusual and creative hacks by white hats, and what these new vulnerabilities might mean for the coming year.