I don’t believe that analyzing every deflection down to perfect empirical proof is a reasonable goal. But I think everyone can agree that you are deflecting calls and you are completing support transactions at a lower cost than in other channels. The question is how many calls are deflected and how many transactions completed. And I think some consensus will emerge about ratios of visits to deflections, etc., as more companies measure this for themselves and share what they have found.

A few examples (and hopefully these will be useful to Matt too):

One approach quantifying about support ROI is based on visits. In the book Groundswell, Josh Bernoff and Charlene Li looked at Dell’s support forums. Dell told them that 7,000 of Dell’s customers log on the to the forums each week, and that exit surveys tell them that “between 20 and 50 percent say they got an answer to their question” (p. 161).

For its generic ROI formula (p. 162) Forrester adopts 33% as the percentage of visitors who get answers and $10/call for cost per call. (Groundswell (p. 157) says call center costs average $6 to $7 per call and tech support calls cost “around twice that: $10 to $20.”) They also estimate that 5% of the total customer base will visit the forums in a given year.

Another data point comes from Cisco. Several years ago, Cisco did a survey that found that 41% of visitors to their Net Pro forums would have opened a support case if forums had not been available. So that sort of conforms with Dell’s “between 20 and 50.”

A final data point comes from one of our customers. I can’t share the name, but they commissioned a study by one of the Big Four accounting firms which found a roughly 3-1 ratio between visits and deflections.

Another approach is to focus on number of answers in your forums and the number of times those answers are read. (Sounds like Matt may be doing this.) Groundswell (p 161) suggests that it might be reasonable to assume that 1 out of 6 people who read an “answer” solve their problem and don’t have to call the call center.

But of course identifying answers and counting how many times those answers have been read is difficult and time consuming — and of course you’d have to deduct the cost of doing that from whatever benefits your company gets from the deflection.:) As an alternative, it seems reasonable to assume that some percentage of posts in every support forum are answers. If you knew that ratio, you could probably come up with some good proxy ROI figures by just looking at total posts and total reads. I’m not sure how they determined this, but Groundswell says that “somewhat less than 1 million” of the 4 million total posts on the Dell forums can be characterized as “answers.”

One out of four seems high to me. The average thread depth in a support forum is usually between 2 and 3 posts, meaning that somewhere between 1 and 2 million of Dell’s posts can’t be answers, because they are questions. Which means that 1 out of 2 or 1 out of three of all replies are answers. (Check my math here, but I believe I’m right.) At any rate, I don’t have a more better data point on this.

Anyhow, I offer the above because it’s often useful to have external sources to validate whatever internal analysis you do. Hope this is helpful!

I am working on a method that uses a by-hand evaluation of frequent samples. Posts are evaluated and sorted into categories. Since our forums are for support, a question that is clearly answered is worth the cost of a phone call to an agent. The challenge that remains is to value the post based on views — who else decided not to call because they found the answer?

It’s much more complex than this, but I think a careful system of this kind can move us much closer to an ROI for support forums. It has the additional value of giving the forum operator (me) a good excuse to dive deep into forum interactions. Reading all those posts in order to value the sample is a great way to gauge the community’s general health.

The halo effect of providing this community to people probably can’t be measured (please prove me wrong) but it probably dwarfs the cost avoidance factor.

On the specific metrics mentioned, you need to identify both the production you are trying to measure and ways of measuring it effectively; this can be a real challenge on a community. One person mentioned tech support calls, there is no effective way to measure this using metrics inside the community alone. Counting incidents of advice will not work, good advice will often stand and be reread over and over again. And you can’t count page views, a page view equals activity for community reporting, but doesn’t necessarily mean a tech call was skipped by the viewer. You can also measure transfer, links to a piece of tech support advice or repostings and these are good numbers but difficult to place a monetary value on.

When measured over time a community that is committed to building a tech support identity, worked to identify product gurus and include them, consistently checked tech support comments on its forums for accuracy, and marketed itself as a hub for tech support conversations will have a measurable effect by lowering the aggregate volume of tech support calls over time. This article highlights the effect on Intuits community: http://tinyurl.com/lzx46d

This is an extremely focused community concept, which I have mixed feelings about (I think communities can be flexible and serve multiple tasks simultaneously) but they got all the basics right. They had a strong concept, built a plan, worked on execution over time, worked closely with the real world department in the company corresponding to their virtual campaign, and achieved strong measurables over time, not only in community numbers, but in corporate expenditures.