Over on Debbie Wilson's blog, there's a good entry that raises the question of the role of product demonstrations in evaluating technology solutions. Even though Debbie tackles it from the role of analyst, I think there are some lessons and broader discussion points to come out of it. For one, consider the role of references in decisions as much as a demonstration and RFP response. However, the best references are not necessarily the ones provided by a vendor -- they often come through your own network. If a vendor provides them, make sure you ask as many open-ended questions as possible to gauge the level of enthusiasm in the response. "Damning with faint praise" should damn a vendor off your shortlist. I often request references from vendors but rely on industry (or channel) connections when possible to do my real homework.
But back to the topic at hand -- the value of demonstrations. All too often, unless the analyst or prospect orders up the demonstration himself or herself, requesting their own demo scripts and scenarios, vendors will attempt to pull the shades over the shortcomings in their solutions (or at least provide basic whitewash to areas where functionally, they can't quite compare to others). In other words, be prescriptive about what you want to see. In a follow-up to this post, I'll provide some specific ideas about how you can drive vendors to provide demonstrations that will most closely mirror your own environment versus what they want to show.
Overall, how should demonstrations fit into your evaluation of a product? In short, as Debbie notes, "If you aren't really sure what a solution does, a demo is a great way to get, at a glance, the highlights of functionality. Often, a picture is worth a thousand words." Moreover, "A demo can also give you a rough idea of how aesthetically pleasing a solution is to use." This last point is essential. Business users must prioritize usability alongside absolute capability in the case of solutions that will touch more than a handful of users. For example, when it comes to spend analysis (not broader spend visibility), I'd personally want the most powerful solution out there, even if it looked like a relic of the client/server era and required an IQ of 120+ to work with. But for eProcurement, sourcing, contract management and other areas -- not to mention broader spend visibility and related reporting -- ease of use matters, potentially uber alles. And if you don't like what you see during a demonstration, I'd scratch the potential vendor off your shortlist as quickly as possible.