12:03 PM
Connect Directly

Sentimental Favorite: Data Mining on Twitter

Could carriers find value on Twitter and social networks as it relates to mining unstructured data?

Sitting in SAP's New York offices on Wednesday for a press conference announcing the launch of Business Suite 7, I couldn't help but be reminded of a blog written just a day earlier by I&T editorial director Kathy Burger.Kathy, who had just created an account on (I have an account too, by the way: nconz), a micro-blogging site, was wondering about its possible business uses. Could a media outlet like I&T use it to find potential sources or bring more readers to its Web site? Could an insurance carrier use Twitter to interact with customers and producers? "I did a search on Twitter for 'insurance' or 'insurance companies' and found mostly individual complaints and concerns about their individual experiences with the industry," Kathy wrote.

It was that last line that I was reminded of at the SAP press conference. In a product demo, Ian Kimbell, VP of business process validation, SAP showed the audience SAP's "sentiment engine," which allows a company to see what users of a site like Twitter (which was used in the demo) are saying about it.

The sentiment analysis is accomplished using the SAP Business Objects Text Analysis product, Rick Fleischman, Director, CRM Solution Marketing, SAP AG, tells me in an e-mail. "It is part of our Business Objects portfolio and provides a sophisticated ability to mine unstructured data to extract entities, persons, organizations, sentiment, etc.," Fleischman explains.

The rule-based linguistic engine can be configured by customers for specific scenarios. "We plugged this engine (SAP Business Objects Text Analysis) into CRM and fed Tweets to it (via Twitter APIs). The engine parses each of the tweets and extracts the sentiment of the conversation from it," Fleischman says.

In the case of the demo, Kimbell used the sentiment engine to analyze "tweets" (Twitter posts) relating to a generic GPS device. The engine essentially aggregated the various mentions of the device and identified trends. In this case, it was determined that many Twitterers (I'm not sure if that's the preferred nomenclature. It could be Twits.) disliked a certain aspect of the device's design.

A similar application might work for insurers, particularly in the area of customer service. As Kathy pointed out, many insurance mentions on Twitter involve complaints and concerns. On an individual basis, those complaints and concerns represent anecdotal evidence at best. If aggregated though, a carrier might be able to identify trends. Perhaps there are specific areas of its customer service or claims operations that are lacking or causing an inordinate number of complaints.

Here's what ZDNet's Editor-in-Chief Larry Dignan (Who, now that I've seen his photo, I'm pretty sure was sitting next to me at the press conference. Hey Larry, wasn't that reporter sitting next to me laughing a little too loud at Ian's jokes?) had to say in his blog post on the sentiment engine:


While Kimbell's demo, which was quite entertaining, doesn't reflect what enterprises are actually doing it does show an increasing amount of integration with social networking tools. If corporate data is merged with the anecdotal tips from customers and partners there could be real insight.

This Holy Grail of insight is what a lot of vendors-Salesforce, Oracle and SAP-are chasing.

Whether enterprises actually use Twitter remains to be seen, but it's not too often you seen an enterprise planning demonstration with a Twitter plug.


East Hartford-based Full Capture specializes in data analytics and the mining of unstructured data. With regards to engines like the one SAP demonstrated, Full Capture CTO Bill Nadal says one key is to avoid false positives. For instance, if a financial services firm is mining Twitter or Facebook, the world "rollover" means two very different things in the context of an auto insurance claim or a 401k.

"In our view, a 'sentiment engine' such as released by SAP, is a very domain specific model targeting a specific context," says Nadal. "Without elaborating on the legal and privacy issues SAP, Facebook and Twitter face with mining this type of data, the analysis of this data will produce many false positives without some hard work in building specific customer vocabularies and contexts and integrating these models into the results. The sentiment engine might operate at a simple level of searching for a company's name or product, and this will have some value in this case. But the true value comes from more advanced levels of semantic analysis."Could carriers find value on Twitter and social networks as it relates to mining unstructured data?

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: This comment is waiting for review by our moderators.
Current Issue
Security Operations and IT Operations: Finding the Path to Collaboration
A wide gulf has emerged between SOC and NOC teams that's keeping both of them from assuring the confidentiality, integrity, and availability of IT systems. Here's how experts think it should be bridged.
Flash Poll
New Best Practices for Secure App Development
New Best Practices for Secure App Development
The transition from DevOps to SecDevOps is combining with the move toward cloud computing to create new challenges - and new opportunities - for the information security team. Download this report, to learn about the new best practices for secure application development.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
Published: 2017-05-09
NScript in mpengine in Microsoft Malware Protection Engine with Engine Version before 1.1.13704.0, as used in Windows Defender and other products, allows remote attackers to execute arbitrary code or cause a denial of service (type confusion and application crash) via crafted JavaScript code within ...

Published: 2017-05-08
unixsocket.c in lxterminal through 0.3.0 insecurely uses /tmp for a socket file, allowing a local user to cause a denial of service (preventing terminal launch), or possibly have other impact (bypassing terminal access control).

Published: 2017-05-08
A privilege escalation vulnerability in Brocade Fibre Channel SAN products running Brocade Fabric OS (FOS) releases earlier than v7.4.1d and v8.0.1b could allow an authenticated attacker to elevate the privileges of user accounts accessing the system via command line interface. With affected version...

Published: 2017-05-08
Improper checks for unusual or exceptional conditions in Brocade NetIron 05.8.00 and later releases up to and including 06.1.00, when the Management Module is continuously scanned on port 22, may allow attackers to cause a denial of service (crash and reload) of the management module.

Published: 2017-05-08
Nextcloud Server before 11.0.3 is vulnerable to an inadequate escaping leading to a XSS vulnerability in the search module. To be exploitable a user has to write or paste malicious content into the search dialogue.

Dark Reading Radio
Archived Dark Reading Radio
In past years, security researchers have discovered ways to hack cars, medical devices, automated teller machines, and many other targets. Dark Reading Executive Editor Kelly Jackson Higgins hosts researcher Samy Kamkar and Levi Gundert, vice president of threat intelligence at Recorded Future, to discuss some of 2016's most unusual and creative hacks by white hats, and what these new vulnerabilities might mean for the coming year.