A Flawed Way of Sourcing Information? Gartner's Two-Week Fire Drill and a Magic Quadrant Lawsuit …

- October 22, 2009 3:10 AM
Categories: Spend Management |

This is not a topic that I like to write about, nor is it one that I think many practitioners and consultants who read Spend Matters and do technology selections — both for themselves or their clients — should spend too much time on. But someone has to call out just how ridiculous the analyst vendor evaluation system has become. Not to mention the onerous time requirements that one analyst firm is putting on vendors, causing fire drills and ultimately distorting the quality of their own analysis in the process. The topic I speak of in this regard is the most recent Gartner Strategic Sourcing Magic Quadrant information request that went out to technology vendors this week. On the surface, this might sound like a good thing — after all, it makes sense to have more objective comparisons out there in the market. But in reality, it’s anything but. Let me explain.

I recently had the chance to help out an old friend in a completely different technology area with a response to a Gartner information request for a similar competitive report. He engaged my services to help refine — perhaps massage is a better word — his firm’s Magic Quadrant response and overall analyst pitch. Now this is not something I like to do — nor is it something I’ve done since my time at FreeMarkets. But he somehow convinced me to lend a hand and since I did not have a conflict of interest with those I cover in the Spend Management world because his was such a different tech universe, I decided to help my old colleague out over the course of a couple of days. It brought back memories, let me tell you.

For one, his team had done a great job answering the questions with perfect honesty. But there was no story weaving the responses together, let alone did they understand what they were going up against (namely big competitors also participating who do not break out their financials and company information elsewhere at a level of detail that allows them, in the case of the MQ, to “spin” their numbers as creatively as possible). Moreover, it was clear the analysts were trying to trip them up in certain areas, but this provider did not have enough experience to know how to avoid these standard traps (which have nothing to do with merit, mind you).

Now granted, much of the MQ process is based on customer references. But we all know on an overall basis, such rankings are based on subjective components based around what criteria you ask, how you analyze it and the ultimate weight you give it. And it’s here I believe that the entire process breaks down. But it does so even more when you force vendors to fill out responses in a matter of days — not to mention forcing them to recruit half a dozen customer references (or significantly more, depending on Gartner’s defined criteria based on number of overall customers) in such a short period of time.

Having no empathy for vendors — let alone customers, because they’ll be the ones stuck with ultimately using an analysis with a highly sub-optimal research process — this is precisely what Gartner did earlier this week when it sent out a request to companies to fill out a complete survey response in approximately a 10-day period (these surveys can take 50+ hours to sufficiently complete depending on the level of detail they ask for). And given that Gartner is asking for responses for “spending analysis, e-sourcing, contract management, [and] supply base management” I suspect it might take the diligent product marketing or analyst relations person even more time than this to find and fill in the appropriate details given that this is really multiple quadrants in one. Moreover, the effort is compounded by the good will expended recruiting dozens of customer references in such a short period of time — which I’m guessing, based on the email Gartner distributed to vendors, will be an average of 15-25 customer references each.

Personally, I think we need to stop for a minute and ask why Gartner is pushing e-sourcing, spend analysis, contract management and supplier information management vendors up against such a tight time line. The other recent MQ project I loaned my time to gave vendors well over a month to gather and submit the requisite information. Because of the short-time frame in this case, there’s no question vendors will be stretched thin filling out this information (not to mention the fact that they’re also currently filling out similar requests from competing firms who are also updating their research in this and related areas, but allowing more time). Given this, I suspect the results will end up flawed, representing a fire drill based around who had the most time to fill in a response and who is lucky enough to have enough customers not currently going through a renewal process who will serve as rapid-fire references. All of this is further proof to me why traditional analyst vendor comparisons are a poor reflection of actual vendor applicability and usefulness to a given customer requirement.

I suppose it is possible for vendors to get even when it comes to this nonsense. But whether or not one vendor who is suing Gartner for what they claim is a biased and flawed MQ research process is successful in the courts remains to be seen. Personally, I’d love to be on the jury in this particular case. But if that’s not possible, I’ll be happy enough sharing the details of how the research works in the case of the Gartner Strategic Sourcing Magic Quadrant to hopefully disuade practitioners from putting too much stock in the results.

Jason Busch

* Required fields  [email address will not be published]

*