I suspect you are right, that “community” is indeed the key element here. One question we have been asking ourselves regarding the “badges” we want to issue to users of the Mozilla/P2PU School of Webcraft is, “should badges be tied to participation (in a course)?” Tying them to a course, or other form of participation in the community, reduces the potential for the system to scale – but greatly improves our ability to say meaningful things about users’ skills and competencies, and reduces the opportunity for gaming the results.

Your title “I don’t need a certificate to beat you in chess” has got me thinking. First it reminded me of stories (true or not) of people passing themselves off as doctors. In this scenario, a person may have some of the skills, may even be brilliant at some elements of medicine but may be critically lacking in other knowledge. The flip side of this is a certified Medical Doctor who hasn’t kept up to date for years with changes in medical practice and research.

Both scenarios are potentially dangerous to patients. So what criteria could you establish that would address both of the above kinds of danger. Perhaps community recognition would do it. Being recognised as an ongoing participant / learner / mentor within a community.

If community recognition is the key, how do you establish a mechanism for that recognition? Is “participation” a minimum requirement? Are we required to participate in order to be recognised?

And if that’s true, what kind of participation? And how do you prevent that metric from being gamed? Google’s pagerank and Ebay’s seller rating are proxies for relevance and trust. I wonder if there is a proxy for competence and knowledge.

Sorry, no answer yet. Thinking out loud.