Aberdeen: Why Quality Control Matters in Benchmarking

Thanks to some friends who pointed me in the general direction of "Aberdeen's largest" survey report ever, the 2007 Aberdeen Executive Report (the link to download the free report is at the end of the announcement), I had the chance to remind myself how the quality of data matters more than the absolute numbers when it comes to benchmarking and research. According to the executive summary from the report, this work is a "compendium of information that spans five years, two million respondents, 644,000 organizations, and up to the minute feedback from 3,670 survey respondents." That's a lot of respondents. But is there really wisdom in crowds? Decide for yourself and consider their findings.

For example ... have 62% of companies really adopted spend analysis technologies? What about 53% implementing contract management technologies? And have 36% really embraced outsourced procurement? In the Fortune 500, these numbers might seem high, but given that Aberdeen featured responses from 644,000 organizations, I'd say that there must be some pretty sophisticated small and middle market organizations out there. Or, as I suspect, Aberdeen's data quality control is not as good as it needs to be.

After all, if these numbers told the actual story of the market, then the spend analysis, contract management, and procurement outsourcing sectors of the market would each be worth well over $1 billion each per year right now -- or maybe 5x or even 10x the number when put together. So much for the intelligence of crowds! Maybe "2 million respondents from 644,000 separate organizations" can really be wrong.

Another pair of eyes who also looked at this same data suggested to me a joke that sheds some light on the situation here:

Question: What do you get when you cross marketing services with market research?

Answer: A demand-generation opportunity in search of a technology area to survey.

Not very funny, my friend said. But neither is the data -- especially if you're using it to make business decisions and it proves wrong. Seriously, if you really want to look at the numbers, seek out sources of information which you can put some real stock in. Personally, I've come to feel comfortable with AMR and Hackett end-user survey data. But decide for yourself. Also, if it's market sizing and forecasting you're after, check out AMR's new marketing sizing report for procurement and sourcing (registration and membership required).

Jason Busch

Discuss this:

Your email address will not be published. Required fields are marked *