For a long-time, I've harbored a secret ambition to do deep-dive vendor comparison reports in a totally different manner than the flawed analyst rankings (quadrants and waves) that I spent a few years trying to influence earlier in my career. One approach is to follow the KLAS ranking model in healthcare technology, a ranking methodology that relies entirely on surveying customers. Aberdeen is finally getting around to try adapting this model to the space and I wish them the best of luck with it (I suggested the idea to them four years ago). But for Spend Matters, even this approach feels too old school analyst-esque. I want to try something new and I also don't want to get involved in rating vendors on a comparative scale per se.
To me, what's more interesting is a deep dive approach that combines true evaluation expertise with actual, real-life based scenarios. To this end, I've decided to start a Spend Matters bake-off series. At this point, we're planning two (possibly three) for the year. The first will focus on sourcing optimization, as prompted by a post that drew quite a bit of controversy late last year. My evaluation and business partner in crime for Spend Matters bake-offs will be Paul Martyn, a former CombineNet executive (and before that, a former chef). We will also invite a guest expert to join us in evaluating each bake-off. For the optimization evaluation, we are excited to have Gail Hohner join the team. Gail has fourteen years of experience designing and implementing optimized procurement auctions for Fortune 500 sourcing groups. She designed and ran the first-ever optimized logistics network auction in Europe and was part of the co-founding team that launched Freight Traders Ltd. She has guest lectured at universities and business schools, participated in expert panels at conferences and is a co-author of Combinatorial Auctions (MIT Press).
How will the bake-off work? Our concept at this point in time -- subject to further refinement -- is to review providers based on their participation in a series of RFPs for different sourcing optimization scenarios. This will be done both offline and online and will include demonstrations. We will then evaluate vendors based on the RFPs they respond to and then roll-up these results into larger case studies based on the strengths, weaknesses and approach of individual providers to solving real-world optimization needs. We will compare providers in relation to an absolute standard rather than in comparison to each other (which in our view is one of the evaluation and design flaws of traditional grading mechanisms in vendor evaluations).
The finalists from the RFP rounds will be invited to Chicago for the live one-day "bake-off". The bake-off will actually take place in more than one way. Given Paul's and my love for cooking, and our past lives spent running a fine-dining restaurant (in Paul's case) and tending bar, serving as prep cook and washing dishes (in mine), we decided to add a true "bake-off" component to the event. To this end, we will not only ask those participants that make it to the final round to demonstrate certain capabilities in a compressed timeframe -- they will also enjoy a dinner on us, with significant audience participation (and possibly even some foodie competition) the night before. Incidentally, for participants wary of sharing too much IP with competitors during the optimization part of the bake-off, the final round will take place in a physical environment that will create privacy protections during actual software demonstrations and configuration. Last, participants who make it to the final round will be asked to pay a nominal fee to offset our logistics cost for holding the event in a downtown Chicago venue (though Paul and I are happy to cover food and wine costs for the group the night before).
This post will serve as the formal announcement of this first Spend Matters bake-off. We intend to invite the following vendors by email: CombineNet, Trade Extensions, Emptoris, Iasta, Ariba, BravoSolution, Oracle and SAP. Two of these providers have already agreed to participate. We hope others will as well. And we welcome entrants from other providers, too.
At the end of the bake-off we'll make the RFPs we've designed, elements of the vendor responses and our assessment of each vendor based on the information (responses, presentations and demonstrations) available. Throughout the process we will keep readers up to date with our progress. Stay tuned on this note. We are, of course, interested in hearing from you on the criteria we should include in our RFPs and welcome your thoughts on some real-world scenarios worth including in this inaugural bake-off.