Can We Have Some Metrics On How Effective Metrics Are?
from the seduced-by-numbers dept
There’s just something about “metrics.” I spent Monday at the Word of Mouth Marketing Association’s Research Symposium, which was supposed to be all about learning how to track the impacts of word of mouth marketing attempts. Of course, that makes an implicit assumption: that you can, need or want to track those things in a simple manner. It seemed like the key point made by many of the attendees (mainly from big companies) is that they need to figure out how to translate “WoM” campaigns back into the common terminology and “metrics” they use in traditional marketing in order to convince CEOs and marketing VPs that word of mouth campaigns make sense. Metrics was the word of the day, with multiple people saying that without metrics, word of mouth marketing was effectively pointless. There’s just one problem with this: metrics can often be worse than useless. Metrics are only as good as what they’re measuring, and they’re often measuring either the wrong thing, or something that can easily be gamed. A month ago, Joel Spolsky had a fantastic post on the ridiculousness of metrics, and how they’re often used mainly by consulting firms to convince big companies to funnel money into the consultants’ bank accounts.
With all of that in mind, it’s not that surprising to see a story noting that web analytics firms (who have been at this for quite some time, and you would have thought had the kinks worked out) are revising the traffic certain sites got downward, often by tremendous amounts. Apparently, the website Entrepreneur.com didn’t have 7.6 million visitors (as reported), but just over 2 million. That’s not a small error. The problem, it seems, is that the site put news content into unwanted pop-up ads, often delivered by adware on lots of computers. In other words, the pageview may have happened, but it wasn’t wanted, and it’s quite likely that it was closed before it even loaded (perhaps with anger). In other words, it was a metric that was gamed. Keep this in mind when you hear silly reports like MySpace passing Yahoo in pageviews according to various metrics companies. As the article linked here notes, part of that may be due to Yahoo shifting their web mail app to AJAX, making it much more efficient and user friendly — but requiring fewer page views. Equally responsible, however, would be MySpace’s awful design that requires extra page impressions to actually get anything done. In other words, in focusing so much on the “page views” metric, the incentives encourage bad design. The companies that win the metrics game are the ones who are designed in ways that upset users, rather than those that make their service better, faster and more efficient. Normally, that would be a clear indication that the metrics are all wrong — but very few people seem to care. They just continue focusing who is ahead of whom on the list — and plenty of bad decisions continue to get made because of this blind allegiance to metrics.
Comments on “Can We Have Some Metrics On How Effective Metrics Are?”
typo
Love reading the posts Mike.
Typo in the top section.
‘metrics can often worse than useless’
Re: typo
Typo in the top section.
Oops. Fixed. Thanks.
Sad to say, this is neither suprising nor newsworthy.
Measurable statistics will always outweigh intelligence, design, and accessibility.
Nice post with good articles/sources cited… however, again, this isn’t newsworthy… just common sense and something that is to be expected.
Metric
When I read the title I actually thought the article would be about the advantages/disadvantages the metric system of measurement has over the Imperial one.
When in fact it turned out that consultants have just latched onto a new word, remember when the word Paradigm was so popular a few years ago? Or when people started adding e and z’s to words that had no business having them?
It was stupid. Still is.
duh!
MySpace gets paid per ad impressions, of course they’re going to maximize those impressions.
Yahoo gets paid per click, thus they want to minimize page loads (no sense wasting extra bandwith.. )
The only metric I care about lately is the number of unique visitors I get. If it’s growing, things are good. If it’s shrinking, things are bad.
The other good metric is activity. what percent of unique visitors post comments, click ads, buy stuff, or “do” whatever there is to do on the website.
Metrics v. faith
The need to have metrics for everything is driven more by fear of making a mistake than by anything worthwhile. The direct, quantifiable effect of word-of-mouth is difficult to measure because it impacts not only page impressions but the visitor’s impression of the company. If a credible source stops by your desk and tells you the new James Bond movie is definitely worth seeing, you will probably go see it. But how can that impact be measured? It can’t, but it doesn’t mean it didn’t have an effect. It may have had more effect than all the TV and newspaper ads you’ve seen.
The accountants and business school types are taught that if something can’t be measured it can’t be managed. But WoM isn’t completely manageable. If it’s going to work, it has to take on a life of its own. That’s really what blogging and social media are all about — getting away from the management and manipulation in order to encourage real conversations betwen real people.
Much of marketing is like going to church. If you believe going to church has value for you then it does. But you can’t measure that value or put it in a spreadsheet. The more people try to measure the value of most types of marketing (direct marketing excluded), the less they’ll focus on actually delivering anything of value.
Gaming the Metric
Usually we can’t measure what we’d like to. An Ad exec wants to know that $X spent on this marketing tool will return $Y in sales. He’d like a site that was so cool and so compatible with his product that every visit results in a sale, and he wants the site owner to work with that as a goal.
The reality is that he can only measure page views, and estimate how page views convert to sales (either by direct click-through or later visits to the sales site). So we measure and pay on page views, and site operators have the goal of increased views, no matter how bad the content they have to build to get them.
Metrics are good only insofar as they measure the thing we are really interested in.
Measurinfg metrics...
First of all, this statement is a little flawed:
I’m sure Google can come up with excellent metrics on something or another, and people liove them. Perhaps you were referring to companies that rely solely on pageviews as a metric?
For the rest of it, I think you are absolutely right that metrics can be manipulated and WoM is hard to quantify, but the nature of business requires some attempt at quanitification. Social networking etc is great for meeting people and stuff but until a reliable way to equate the effectiveness of social networking and WoM to predictable consumer patterns (or opinions) it’s a tool that most businesses will not invest too heavily in.
Can you imagine some marketing guy going to the CEO of a company and saying “Hey, our product is going to sell lots of units because a lot of people are blogging about it and response has been favorable.” The first questions the CEO is going to ask are 1) How many people are blogging about it and 2) What does favorable response mean in terms of predicted sales? Based on that answer money is allocated, product numbers are set, and schedules are made.
If the vast difference between the WoM generated by “Snakes On A Plane” versus the actual ticket sales is any indication, corporations are going to be very shy about utilitizing WoM as a metric.
acronyms.
WoM is more difficult to say than word of mouth. more vowel syllables constituting large tongue and mouth movements. If these asspandas are looking for efficiency, they certainly arent thinking efficiently. Then of course theres www and world wide web. measurable results of statistical analysis are cool. turning “dude I got a dell” into measurable data may be good for some marketing techniques. But please, PLEASE don’t label it the DIGAD metric. ok. more coffee.
Good metrics are difficult
Metrics are tools, plain and simple. The use, mis-use, or ab-use of them is where the trouble begins.
Gaming of metrics will occur, and it is up to the consumer of the metric to critically view the information presented by the metric and interpret how to best respond.
The odometer of a used car for sale is an example of a metric. You can use it to estimate the level of use/abuse the car has taken. The metric can be “gamed” by shady dealerships, although development by auto manufacturers is making this more difficult, and services like carfax can help to support whether the metric is accurate.
Creating good metrics is a difficult activity, and presenting the data calculated by the metrics is also somewhat of an art. Joel Best’s book, Damned Lies and Statistics, addresses this topic better than I can.
Saying metrics are “rediculous” is short sighted. Joel Spolsky’s article addresses the difficulties in using metrics to manage software developers. While this may be true for that industry (and I’d argue he’s using metrics that are too broad and subject to gaming), it’s not universally true.
Metrics can be useful. Consider calories per serving, miles per gallon, P/E ratios, and processor clock speed. All may help you in making a purchasing decision – it’s a matter of how you use or ignore them.
Metrics are numbers. Numbers are neutral. The creation of the metrics and (ab)use of them is where the trouble begins. But I repeat myself.
And as for ignoring metrics, consider the risks next time you have a pair of flashing red or blue lights behind you as you are driving the highway. Was that “miles per hour” metric-producing speedometer useful or “rediculous”.
all metrics are equal (yeah right) but some are mo
Metrics are still useful. It just depends on which metric you are looking at. Page views are probably worse than useless as you’ve mentioned but unique visitors are still useful in measuring growth.
Metrics indeed
Great thoughts. I agree that metrics are needed but that closer scrutiny is needed, after all, statics of any kind can (and are) manipulated all the time. It’ll be interesting to see how improvements such as AJAX-enabled functionality (and the advent and implementation of Adobe Apollo-style rich internet applications) will impact web metrics.
We’re certain to see big changes in the way we measure (and what we measure) in the future.
Thanks for thinking out loud about the metrics issue.
Scott Abel
TheContentWrangler.com
comment on article
Good article