Fireman’s Tests Load Capacity

Implementing an application without certainty of its total load capacity is a little like serving a meal to an unknown number of guests. During the development of some n-tiered applications for its agents and insureds, management at Fireman's Fund Insurance (Novato, CA, more than $4.7 billion in gross written premium) realized that it "didn't know how many users could concurrently use Fireman's applications," says Robert Mitchell, director of e-business infrastructure, Fireman's Fund. "From an e-business point of view, that is very dangerous. It's hard to predict concurrent use of an application, so you better try and test capacity before production."

Narrowing the Field

In late 2000 a team representing Fireman's infrastructure support, applications and e-business groups began evaluating providers of products that would test applications at different load capacities. Concerns about compatibility with Fireman's Microsoft (Redmond, WA) NT- and Windows 2000-based development environment and its IBM (Armonk) AIX and Websphere testing and production environments, as well as vendor stability, helped narrow the list of prospects to three. Mercury Interactive (Sunnyvale, CA), which Mitchell describes as a leader in the client/server and n-tiered testing space, was the ultimate choice.

Initially, Fireman's ran a pilot as part of a formal investment analysis that would eventually prove the project could generate an appropriate ROI. A Mercury consultant was brought in to work with Fireman's e-business Test and Certification Team—which, according to Mitchell, was very new at the time. An ASP application was chosen to test Mercury's Winrunner and Loadrunner tools' capabilities.

The actual technical implementation of Mercury's tools was completed within 38 days, reports Mitchell. "The process of making sure that you have the competency of the people who use the tools takes longer," he adds. "That was a major focus of the deployment." Because Fireman's test engineers had limited experience with Mercury's toolset, the carrier brought in Spherion (Fort Lauderdale), a consultancy with Mercury tool expertise. "It took about six months of mentoring to get the engineers to the point where they could efficiently create automated scripts and quickly run stress tests against applications," says Mitchell. Automated scripting is used to mimic the keystrokes and mouse actions a user takes when navigating through Web pages and filling in forms in order to complete business transactions. "Scripts are created based on the requirements and use cases that define an application," Mitchell explains. "You can then run the script to simulate many users interacting."

Looking back on a process that he describes as "pretty clean," the only snag Mitchell recalls was a technical compatibility problem. "In some cases Fireman's had a certain rules-based engine that was harder to use the scripting tool against," he says. Luckily, the manufacturer of the non-compatible engine was a partner of Mercury's and the two technology providers fixed the problem, which highlights another advantage of the solution. "Mercury has many partnerships," Mitchell says.

Although the benefits of the tools, which cost Fireman's $500,000 (including implementation), are hard to quantify, they have been numerous. "They're largely intangible, but very important from a quality standpoint," says Mitchell. For example, the Mercury technology is saving Fireman's from having to form contractual relationships with ASPs whose applications don't run in production, he says. Most fundamentally, says Mitchell, Mercury's tools give Fireman's the ability to test loads before mission-critical applications are released, ensuring standards of performance. This is important for applications that face internal employees, as well as producers and policyholders. "It goes all the way down to the underwriters," says Mitchell. "If we release applications that fail, people may begin to distrust their whole relationship with us."

Fireman's original project goals have been exceeded. "These tools demonstrated so quickly a value to the applications area," stresses Mitchell. "They were able to address problems, so we could fix them early enough before production."

---------------------------------

Case Study Closeup

COMPANY:Fireman's Fund Insurance (Novato, CA, more than $4.7 billion in gross written premium).

LINES OF BUSINESS: Personal and commercial lines.

VENDOR/TECHNOLOGY: Mercury Interactive's (Sunnyvale, CA) Winrunner, Loadrunner and Active Tune.

THE CHALLENGE: Test load capacity before applications go live.