Ironically, sophisticated IT management tools can create bottlenecks for the very problems they're designed to solve. Such was the case for Sentry Insurance when an outage and subsequent slowdown affected a critical customer-facing application in spring 2010. "Although a resolution team worked on it for several months, intermittent errors still remained," recalls Jeff Sanner, director of application services for the Stevens Point, Wis.-based insurer.
Consequently, Sentry ($11 billion in total assets) began searching for a better way to manage application performance. "We wanted a tool that would help identify potential issues proactively and assist with pinpointing what ... was causing the issue," explains Shad Struble, senior technical architect for Sentry. "And we hoped for a tool that could drill down into the code level to identify issue contributors."
During April and May 2010, Sentry researched options and developed a short list of five vendors. A month later, two finalists were asked to perform back-to-back proofs of concept onsite.
"We supplied two production situations for the POC. One included the intermittent errors we were never able to resolve," Struble recounts. "Then, during its four-day POC, dynaTrace [Waltham, Mass.] was able to help locate the issue, pinpoint the problematic code and enable Sentry to initiate a trouble ticket with the responsible third-party vendor," Struble continues, declining to identify the vendor.
After inking a contract with dynaTrace in September, Sentry delayed implementation a month to take advantage of a new release of the software and allow senior developers to receive dynaTrace training. Meanwhile, to obtain the scale needed for an enterprise dynaTrace deployment, Sentry established a small dedicated server farm, some of it virtualized, using commodity hardware.
That environment includes a dedicated Microsoft SQL Server database to maximize information gathered during troubleshooting, Struble notes. "We're very interested in collecting fine-grained, long-term performance data to facilitate capacity and resource planning," he says.
A Good Problem to Have
If there was a challenge during the implementation, Struble suggests, it was a good problem to have: too much under the hood. "Because the dynaTrace solution does so much more than just business application monitoring, we're evaluating all of our tools, including dynaTrace, to determine what types of monitoring should be done by which tool," he explains.
By March 2011 Sentry completed the integration of the dynaTrace software with all critical production systems, resulting in immediate performance improvements, according to Sanner. "One of our policy administration systems involves 50 servers," he relates. "During the first quarter, our highest-activity season, the dynaTrace monitoring dashboards allowed us to identify servers and transactions that were having an issue, take them out of production and, within minutes, resolve the issue -- all without any impact to users."
Other benefits include holistic monitoring of server utilization, transaction rates and transaction response times. Overall, Struble adds, Sentry's IT department has greatly improved its ability to meet previously unattainable service-level agreements. "Root cause analysis for issues occurring in complex, many-tiered systems used to require many days and many teams," he comments. "Now it can often be accomplished with the click of a button."
Sentry plans to roll out and integrate dynaTrace across its various production, development and testing systems. Plans also call for developing more-sophisticated SLAs and creating advanced dashboards for senior management, Sanner says. "In IT we've minimized emergency drills and we're addressing issues before they occur in our production environments," he reports. "Best of all, our customers are more productive because there's less downtime and applications are more responsive."