12:03 PM
Connect Directly

Sentimental Favorite: Data Mining on Twitter

Could carriers find value on Twitter and social networks as it relates to mining unstructured data?

Sitting in SAP's New York offices on Wednesday for a press conference announcing the launch of Business Suite 7, I couldn't help but be reminded of a blog written just a day earlier by I&T editorial director Kathy Burger.Kathy, who had just created an account on (I have an account too, by the way: nconz), a micro-blogging site, was wondering about its possible business uses. Could a media outlet like I&T use it to find potential sources or bring more readers to its Web site? Could an insurance carrier use Twitter to interact with customers and producers? "I did a search on Twitter for 'insurance' or 'insurance companies' and found mostly individual complaints and concerns about their individual experiences with the industry," Kathy wrote.

It was that last line that I was reminded of at the SAP press conference. In a product demo, Ian Kimbell, VP of business process validation, SAP showed the audience SAP's "sentiment engine," which allows a company to see what users of a site like Twitter (which was used in the demo) are saying about it.

The sentiment analysis is accomplished using the SAP Business Objects Text Analysis product, Rick Fleischman, Director, CRM Solution Marketing, SAP AG, tells me in an e-mail. "It is part of our Business Objects portfolio and provides a sophisticated ability to mine unstructured data to extract entities, persons, organizations, sentiment, etc.," Fleischman explains.

The rule-based linguistic engine can be configured by customers for specific scenarios. "We plugged this engine (SAP Business Objects Text Analysis) into CRM and fed Tweets to it (via Twitter APIs). The engine parses each of the tweets and extracts the sentiment of the conversation from it," Fleischman says.

In the case of the demo, Kimbell used the sentiment engine to analyze "tweets" (Twitter posts) relating to a generic GPS device. The engine essentially aggregated the various mentions of the device and identified trends. In this case, it was determined that many Twitterers (I'm not sure if that's the preferred nomenclature. It could be Twits.) disliked a certain aspect of the device's design.

A similar application might work for insurers, particularly in the area of customer service. As Kathy pointed out, many insurance mentions on Twitter involve complaints and concerns. On an individual basis, those complaints and concerns represent anecdotal evidence at best. If aggregated though, a carrier might be able to identify trends. Perhaps there are specific areas of its customer service or claims operations that are lacking or causing an inordinate number of complaints.

Here's what ZDNet's Editor-in-Chief Larry Dignan (Who, now that I've seen his photo, I'm pretty sure was sitting next to me at the press conference. Hey Larry, wasn't that reporter sitting next to me laughing a little too loud at Ian's jokes?) had to say in his blog post on the sentiment engine:


While Kimbell's demo, which was quite entertaining, doesn't reflect what enterprises are actually doing it does show an increasing amount of integration with social networking tools. If corporate data is merged with the anecdotal tips from customers and partners there could be real insight.

This Holy Grail of insight is what a lot of vendors-Salesforce, Oracle and SAP-are chasing.

Whether enterprises actually use Twitter remains to be seen, but it's not too often you seen an enterprise planning demonstration with a Twitter plug.


East Hartford-based Full Capture specializes in data analytics and the mining of unstructured data. With regards to engines like the one SAP demonstrated, Full Capture CTO Bill Nadal says one key is to avoid false positives. For instance, if a financial services firm is mining Twitter or Facebook, the world "rollover" means two very different things in the context of an auto insurance claim or a 401k.

"In our view, a 'sentiment engine' such as released by SAP, is a very domain specific model targeting a specific context," says Nadal. "Without elaborating on the legal and privacy issues SAP, Facebook and Twitter face with mining this type of data, the analysis of this data will produce many false positives without some hard work in building specific customer vocabularies and contexts and integrating these models into the results. The sentiment engine might operate at a simple level of searching for a company's name or product, and this will have some value in this case. But the true value comes from more advanced levels of semantic analysis."Could carriers find value on Twitter and social networks as it relates to mining unstructured data?

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Register for Insurance & Technology Newsletters
White Papers
Current Issue
Insurance & Technology Digital Issue
Innovation? Check. Core modernization? Check. Security? Check. Today's insurance IT challenges don't stump this year's Elite 8.