Back to Hub

How to Interpret SolutionMap Procurement Provider Technology Rankings

06/22/2018 By


When you first lay eyes on a SolutionMap Ranking you’ll notice the “Reading the Ranking Chart” box on the right hand side stating a key — but there’s more to interpreting SolutionMap. Let’s explore the different elements of which you should be aware.

Please note: In the Q2 SolutionMap ranking graphics, providers with an asterisk (*) next to their company name not yet evaluated against Q2 2019 RFI requirements, but rather normalized based on scoring from Q1 2019 publication.

1. What You’re Looking at in Each Persona Graphic

Each graphic has two axes in play. The (y) axis represents the analysts’ score of the vendor’s solution. The (x) axis represents the customer survey score of the vendor.

To further help frame what’s actually represented in each graphic, here’s a helpful note on their creation:

All SolutionMap persona graphics are centered at the average of participating vendor scores across the analyst score (y) axis and customer score (x) axis for that specific graphic and scaled accordingly. Vendor and customer scores, and hence relative placement of a vendor’s bubble, will differ by graphic as each score or response is weighted differently across the different personas.

2. Customer References

You will notice the varying colors of the bubbles on the chart. The colors indicate how many customer references were taking into account for a respective provider. Dark purple indicates a ‘maximum’ and grey a ‘below average’ quantity of references. I’m sure you’re wondering how many references the different bubble colors represent, but there is no single answer to that question. You see, both the providers and Spend Matters continuously solicit references year-round. This changes how many references are considered the average.

For instance, there were nearly 1,000 individual, validated references taken into account for the Q2 2019 SolutionMap rankings. These customer references are valid for 2 years upon submission and manually validated by the Spend Matters team by checking the identity of the submitter and ensuring the responses do not seem suspicious (such as extreme scoring throughout the survey out of line with the average for that provider or multiple submissions from a single IP address).

The greater number of references for a SolutionMap category, the higher the average number of references. For any new release, Spend Matters requires a minimum of three references for a single provider per SolutionMap category in which they participate. (For example, if I participate in Sourcing and Analytics, I need to obtain a minimum of six references. It is possible there is overlap if a single customer uses both modules and selects both of these as applicable in the review survey — I would still need three references for each category but it may be that I only need three customers to get them if they all use both modules and include them in their review).

3. Assess a Vendor’s Number of Customers, Employees and Revenue

Fall 2020 Update: In past SolutionMap charts, a vendor’s “bubble size” solely indicated its number of customers. While that demonstrated some level of market penetration, it did not adequately convey a vendor’s experience and resources to handle complex global deployments. Going forward, the bubble size calculation methodology will consider the following:

  • market scale based on the global number of customers
  • number of full-time employees focused on the relevant solution(s) at the provider
  • revenue per customer in the relevant solution(s)

These factors were assessed by the Spend Matters analyst team, and SolutionMap participants had an opportunity to validate them.

Note: Bubble color continues to indicate the relative number of validated customer references submitted per vendor, with dark purple representing the highest number of references.

4. The Four Quadrants

In terms of interpreting the four quadrants of the ranking charts: Providers can be recommended by either achieving Customer Value (horizontal axis) or Solution scoring (vertical axis) that is above the average based on the weighted criteria for each specific persona, which are as follows:

  • Solution Leaders are recommended providers with above average analyst scores (upper left quadrant).
  • Customer Leaders are recommended providers with above average customer scores (lower right quadrant).
  • Value Leaders are providers with both above average analyst and customer scores (upper right quadrant).

Emergent Contenders are providers who did not achieve above-average scores (lower left). To encourage participation among the widest range of service providers — especially those specializing in a specific market or sub-process niche — and to simplify how procurement practitioners use the rankings, SolutionMap graphics only feature recommended providers for each persona. Specialists and emerging providers may not (yet) score well enough overall to appear on a SolutionMap graphic but might be worth your consideration. Feel free to inquire to learn more.

5. More on a Provider’s Relative Placement on the ranking graphic

Again, all persona graphics are centered at the average of participating vendor scores across the analyst score (y) axis and customer score (x) axis for that specific SolutionMap category and scaled accordingly. Vendor and customer scores, and hence relative placement of a vendor’s bubble, will differ by graphic as each score or response is weighted differently across the different personas.

You’ll also notice that providers are situated in a different spot on the ranking graphic of each SolutionMap update, which as of 2020 are released twice a year, in the Spring and Fall. That’s because SolutionMap is not static: As solution providers continually invest in R&D and improve their capabilities, some features and requirements which originally yielded higher solution/analyst scoring will see less differentiation with each SolutionMap release. Three possible reasons:

  • Providers did not make additional investments in a functional area and lost ground to competitors
  • Providers did not demonstrate newly added capability to the Spend Matters team during the review process, done twice a year
  • Newly added providers set a higher bar for differentiation and advanced capability

That’s why Spend Matters recommends that existing participants in the SolutionMap process provide RFI and solution demonstration updates every other SolutionMap release cycle, or more frequently in cases of major solution releases or product launches.

The same goes for the customer references scores — they aren’t static either: Spend Matters both encourages providers to obtain customer references year-round and solicits procurement practitioners directly. The more customer references, the more accurately the user experience with the provider is represented. Also, customer references need to be renewed every 2 years (this period was extended to 2.5 years in 2020 due to coronavirus pressures) or they will be removed. New providers will alter the average benchmark and thus the relative ranking of other providers on the chart as well.

Can I request my company be removed from being shown on SolutionMap?”

Your company’s solution will continue to be visibly ranked in our SolutionMap graphics until either:

  1. Your customer score and analyst score drop below the average, i.e., when your score falls into the lower left quadrant, and/or
  2. You have no more active customer references (which age out after 2 years).

If you haven’t yet explored SolutionMap procurement technology service provider rankings, do peruse the site at your leisure and feel free to contact Spend Matters if you have further questions.

Last updated: Oct. 1, 2020