Measuring media coverage is a complicated task. It’s a no-brainer for your team to want both accuracy and efficiency at the forefront of any monitoring and analysis solution, but it can be difficult to find a tool that succeeds at both.
With a plethora of different options to choose from, including many automated solutions that use technology alone to provide media metrics, it’s important to ask – is this analysis something we can trust?
The Problem with Data
For many leading companies, accurate analysis of text remains a problem yet to be solved. IBM has estimated that one-third of business leaders don’t trust their own data, and even more staggering, that $3.1 trillion of America’s GDP is lost due to bad data.
Researching Humans v. Machines
Considering this high risk that inaccurate data has across industries, a recent collaboration with Toyota Motor NA Energy & Environmental Research Group, PublicRelay and students at the Massachusetts Institute of Technology Sloan School of Management (MIT Sloan) Analytics Lab set out to find an answer to the question “can computers analyze media coverage as well as humans?”
After a series of analytical tests against social media postings conducted over the course of 3 months, it was determined that the machine-based analysis could only detect 9% of key messages, identify 20% of appropriate sentiment, and determine 33% of the customer experience.
Conducting the Study
In order to conduct their research, the MIT Sloan Analytics Lab tested various technologies to:
- Understand which topics car enthusiasts are discussing in relation to Toyota’s alternate-fuel vehicles on Twitter;
and - Identify “significant” tweets based on topic inference from above.
The goal was to improve the Toyota team’s understanding of consumer behavior, to improve consumer perceptions and sentiments, identify tweets that demand further tracking or direct engagement, and formulate messaging that Toyota might use to drive social media conversations themselves.
Answering the Data Question
Rather than accurately providing the analysis itself, the students found that the technology was most useful for eliminating irrelevant posts before humans analyze them.
The results of this student project showed that rapidly emerging issues or breaking news were slow to be reflected in the automated technology model—since the machines had to “learn” how to categorize it and this took a fair amount of time. The results also showed that 80% of the machine-learning work was purely data clean-up.
These days, data and analytics are critical to decision making in every business function, and Communications and PR is no exception. But teams need to select solutions that don’t just provide data, rather they provide quality data. And based on the MIT/Toyota study, it seems that computers are not yet up to that task.
Click here to learn more about PublicRelay’s partnership with Toyota and MIT regarding machine analysis.
