How SolutionMap compares to Gartner, Forrester, IDC in rating procurement technology

When it comes to rating procurement technology, what do you trust — your gut or your data?

In a new white paper, Azul Partners and Spend Matters Founder Jason Busch compares the Spend Matters SolutionMap ratings approach to other analyst ratings methodologies, including the Gartner Magic Quadrant (MQ), the Forrester Wave and the IDC MarketScape. The fundamental argument that Busch makes is that the "data" that drives legacy rating models and vendor placement is circumspect in terms of accuracy and predictive insight. And that is not because the analysts at these firms are any less smart or expert than those at Spend Matters. Quite the contrary, in fact. It's the reliance on their very expertise that is creating the problem!

But how can that be?

‘Thinking, Fast and Slow’

We need to turn our attention to the Nobel Prize-winning research of Daniel Kahneman (and Amos Tversky) to understand why. When exploring the approach of other ratings through the lens of Kahneman’s book “Thinking, Fast and Slow,” we see that even the deepest subject matter experts are often as wrong as they are right when forced into a decision model that requires their past knowledge and experience as the foundation for judgment.

In his review of the Israeli military’s process for candidate selection and placement, Kahneman found that having interviewers rate specific, narrow criteria rather than making summary observations was more accurate than approaches based on experience and intuition (or their “gut”). The criteria-based approach helped eliminate bias and led to more reliable ratings of the candidates.

As Busch explores in the paper, traditional analyst ratings approaches require judgment calls on inherently subjective topics vs. rating capabilities against defined criteria and a set scoring scale. If you agree with Kahneman's model, then rating models that rely only on tangible, demonstrated and measured criteria should be more accurate and predictive than the alternative.

Comparing Other Methodologies to SolutionMap

Despite established academic evidence that would call into question the efficacy of the "data" driving their ratings, traditional analyst matrixes are still used every day by procurement, finance and IT organizations to shortlist technology providers. Busch argues that this can lead to various issues — from the wrong vendor selection decisions to reduced ROI, technology investment write-offs and added change management costs.

But why? Busch notes that the data driving each Gartner MQ in part relies on "business-level criteria ratings — of which basic function and technology capabilities comprise limited inputs.” This may force analysts to make the type of macro-level judgment calls that Kahneman would say are no better than a guess in accuracy. In comparison, the Forrester Wave has a "level of granularity in technology and functionality capability scoring" that is comparatively even  lower than Gartner. And when it comes to the IDC MarketScape, the methodology does not even include a granular set of functional or technology scoring criteria alongside business-level considerations, at least based on the published approach.

In short, each of these approaches relies on varying degrees on expert judgment calls.

The Final Word

But expertise has its limits. This leads Busch to opine that the “Spend Matters SolutionMap [methodology] ... is likely to form a more accurate point of reference for understanding comparative technology approaches and demonstrated functional capability.”

Whether you go with your gut or your data on this one, be sure to download the free white paper today for more on how SolutionMap compares to other industry analyst ratings methodologies.

Download Now: "What Can the Israeli Military and a Nobel Prize-Winning Economist Teach Us About Better Technology Decision Making?"

Share on Procurious

Discuss this:

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.