Back to Hub

Jason Busch Turns His Analyst’s Eye on Starting Spend Matters, SolutionMap: ‘We Were Talkers for Too Long’

03/27/2019 By

As Spend Matters is set to turn 15, we asked Founder Jason Busch to reflect on how Spend Matters began and how it has grown over the years.

One development he sheds light on is what sparked the creation of our benchmarking tool, SolutionMap, which was just updated for this quarter with rankings of 58 procurement software providers across 12 categories, like P2P, CLM and Contingent Workforce & Services. The idea stemmed from our analysts always talking about doing more meaningful reviews of procurement solutions than what the market provided — and then they stopped talking and did it.

“Believe me,” Busch said, “we were talkers for too long.”

When you started Spend Matters some 15 years ago, what was your vision for the company?

I started Spend Matters in the fall of 2004. The original goal was to stay alive — as I think with all start-ups — and not have to go back to having a real job, or so that was the plan!

Initially, we did a bunch of custom research, “ghost” white paper writing and strategy work for a diverse set of clients — tech vendors, procurement organizations, consultants and even other analyst firms — simply to pay the bills.

For a number of years, Spend Matters was a hobby in the scheme of things. But it had a small, loyal following that started to snowball as more and more folks heard about it. Along the way, Pierre Mitchell, who was the first procurement industry analyst (at what would become Gartner, AMR Research), joined as our Chief Research Officer.

When did you conceive of the plan to develop SolutionMap and why?

Shakespeare once wrote that “talkers are no good doers.” Believe me, we were talkers for too long. Everyone inside Spend Matters had an opinion about existing analyst comparative frameworks and reports. For years, we talked about how we could do it better. And you guessed it — nothing got done. Remind me never to be a talker again!

Finally, after much brainstorming (I also credit the original thinking to Xavier Olivera, lead analyst, and Pierre), we decided to push forward with an objective vendor ranking model built on “personas” vs. a single view of the data we would collect. I think Pierre came up with fully segmenting analyst ratings from customer ratings, a hallmark of SolutionMap.

Why should a procurement pro trust SolutionMap analysis (versus others)? How are you/your team objective?

To be honest, I have never found other analyst comparisons terribly useful in shortlisting or tech selection. Certainly, their research can be valuable, but not for making actual technology decisions or creating shortlists by comparing business requirements to very specific functional specifications and customer satisfaction metrics. At its core, SolutionMap can (and is meant to) replace the RFI process for companies; the other approaches are just meant to help add companies to include on an RFI distribution list.

At traditional analyst firms, the comparative technology research is not as valuable as the advice the analysts provide on a 1-to-1 basis.

Regardless of the strengths of individual analysts, other evaluation processes are limited. For example, how could anyone possibly evaluate technology in a comparative manner without spending a lot of time with it — and dissecting it capability-by-capability, feature-by-feature.

I also take issue with the ability of analysts — even the best and brightest — to rate a company’s corporate performance and strategy and how that translates to the ability to execute. Even the smartest minds in management consulting can’t judge a company’s ability to execute on the limited insight that analysts glean from this sort of evaluative process. I personally would not trust any analyst evaluative process to accurately derive these insights with the limited datasets available to them from the vendor. It’s hard enough to do this if you have full license to investigate anything during a due diligence process in an M&A situation, let alone when you’re looking at dozens of vendors at once — with only partial information sets.

Moreover, co-mingling customer opinion with analyst assessment creates a less valuable framework for those actually making decisions.

Even though this sounds critical — we are as critical of ourselves as well at Spend Matters. Other analyst firms are excellent firms and worthy competitors. In fact, they are our “mark” to do better than and the yardstick by which we measure our progress.

What is the role of technology analysts? What should it be?

I believe that all technology analysts can and should do is what they know — rate technology for technology’s sake. That’s what we try to do with SolutionMap (and we can always do it better). We spend thousands of hours reviewing and validating detailed RFI specifications (and vendor responses) and on detailed vendor demonstrations where we often ask vendors to “go off script” to show us capabilities on the fly.

The SolutionMap methodology was heavily influenced by Daniel Kahneman’s research. The Nobel Prize winner’s best-known book is “Thinking, Fast and Slow” — an absolutely fabulous summary of his experiments and scholarship. Kahneman, a psychologist by training who received his Nobel in economics, discovered that experts can often issue flawed judgments when they have the chance to interject their opinion in ratings. This research eventually helped the Israeli military decide how to place recruits.

Kahneman found the best approach to matching military recruits to the right jobs was to have interviewers rate responses to specific questions and to never deviate from script. This resulted in the best possible match, correlated with future success.

The analysts do precisely the same thing at Spend Matters. We rate each individual feature, capability or technology component on a scoring scale often unique to each question with a specific guide for each possible rating (for example, a score of 3 on a scale of 5 requires providers to demonstrate very specific capability) — and we allow vendors to do the same, rate themselves. Although we only take the analyst score as final, it provides vendors with a benchmark themselves based on how they view their capabilities vs. how they actually stack up in the market. Vendors can also subscribe to an inexpensive benchmark to see how their final rating for each field compares to the market average.

We also provide a chance to all participants to argue their scores with us on a field-by-field basis during the “appeal” phase. But once this is final, we move straight to publication, and providers do not see the final SolutionMap rankings until they are published. The only appeal that vendors have is based on arguing over tangible, granular scores.

And they can do this every quarter or however often they submit RFI updates (we encourage twice per year).

For 2019, each modular SolutionMap may comprise as many as 300-350 individual ratings. That’s a lot. But some of these fields receive minimized or no weighting in the composite personas — whereas others receive significant weighting. Moreover, these fields represent just 50% of the score — the analyst/functionality score. The other 50% is entirely based on customer survey analysis and benchmarks. We currently have 850+ companies that have peer reviewed solutions in our public database (we maintain a private one as well).

So the question of objectivity is an important one every reader of these types of reports should have. I go to bed every night knowing that the SolutionMap process delivers greater objectivity, transparency and fairness to all those participating than other approaches we have seen.

Other reports are just that — reports. SolutionMap has already become a benchmark for functional capability and customer satisfaction. Users can decide which areas they value and deconstruct the ratings criteria based on what they want to achieve (this also includes solution providers who can, for the first time, really discern exactly how they stack up to peers, not only overall, but on as granular of a basis as they want to see).

Does SolutionMap really matter in the marketplace? You pretty much rank the same companies as Gartner and Forrester do, and they all indicate the same leaders. What makes SolutionMap more relevant?

SolutionMap is less than 2 years old. So does it matter? It’s starting to. It’s certainly influential. We’ve had over 20,000 organizations access the SolutionMap rankings on Spend Matters with thousands of report downloads — along with millions of social media impressions.

Also, we have dozens of organizations subscribing to our member reports and have a goal of conducting 100 individual software selection/shortlisting projects this year using a process we call SolutionMap Accelerator that maps business requirements directly to technology capabilities (complementing or doing away with the need for personas since the Accelerator creates a 1-to-1 match between organization and technology supply market). It’s a faster, better and much cheaper way to do selection work, either directly with Spend Matters or with a consultant you place in the middle (we’re flexible).

Moreover, consultants are subscribing to our member content too — as are investors (private equity, mostly) doing due diligence on the market and specific vendors. So you could say we’re “influencing the influencers” and investment community too — and it’s always fun to see other analyst firms hitting our SolutionMap pages as well.

But there’s really only one measure of whether something like this matters: Is it a sustainable business?

In forensic accounting, it’s all about following the money. In that capacity, we are building something. The first year after launching SolutionMap (2018), I told the team I would sign up for the P&L to build the SolutionMap commercial business with a goal of $1 million in year one revenue. We hit our target.

So yes, we are making a difference. I believe we have a superior methodology.

Ultimately, the reason anyone should care about SolutionMap is because it is useful to them — more useful than other alternatives. I’ve yet to come across anyone who digs into it who does not agree with this statement (even if approximately 25% of those encountering SolutionMap first have questions about the persona-based approach — an education process we are improving upon).

Are you featuring and comparing niche players and market disruptors vs. established providers?

Other analyst firms often set their own criteria for inclusion in comparative analyses. This is usually based on revenue and customer size. So the small, innovative players are sometimes knocked out of the process. Our approach is different. The only requirements are to complete the detailed RFI, provide a complete demo, support our analyst team with follow-up demonstrations and answers to questions, and to have a minimum of three customer references (the average is over 10).

Those providers that complete these requirements are “in” — including both niche players/disruptors and established providers. This creates an open and fair playing field.

If companies buying technology want to knock out specific providers because they are too small, they can do that. They can judge that because our qualitative insight in SolutionMap Insider reports provides information on each provider’s revenues, industries, customers, employee count, etc.

But we don’t want to limit the available supply market from which companies can pick the right technology due to the SolutionMap benchmark process and methodology.

What’s your vision to grow SolutionMap?

Execute, execute, execute. We have a better process and product than anything else in the market today. It passes the test of being less expensive and incrementally better than alternatives. Plus, we gain the distribution benefit of being both a digital media and analyst firm. But without a relentless focus on execution, SolutionMap could become a footnote in the broader annals of procurement technology. And I’m not going to let that happen.

So the growth vision is to continue to “do” — and to get SolutionMap in the hands of more and more organizations worldwide.

Along the way, we will be adding new areas beyond just source-to-pay and services procurement that touch on adjacencies such as finance (accounts payable automation, early payment financing, etc.) and the direct supply chain as well. In many cases, we have the benchmark scoring criteria and fields to rate already completed in these areas — we just need to invite specialized providers to participate.

Read more about yesterday’s update of SolutionMap here: SolutionMap: 58 Procurement Software Companies Ranked (Q1 2019 Update)