Why Competency Self Assessments are essentially flawed beyond redemption

I'm working on an evaluation to determine if a training programme for senior government managers makes a difference - In the competence level of the managers, and in the service delivery they are able to produce within their work context. We are tracking a wide evidence base about all of the participants, but the client is insistent that a competency self-assessment be included. We agreed, on the condition that this one piece of evidence will be used together with all of the other evidence we will be collecting throughout the study. The value that the competency self-assessment will add, is something we have debated in the team. The following entertaining post by Errol Morris, however, pretty much sums it all up:

http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/


David Dunning, a Cornell professor of social psychology... wondered whether it was possible to measure one’s self-assessed level of competence against something a little more objective — say, actual competence. Within weeks, he and his graduate student, Justin Kruger, had organized a program of research. Their paper, “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments,” was published in 1999.

Dunning and Kruger argued in their paper, “When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it. Instead, …they are left with the erroneous impression they are doing just fine.”
It became known as the Dunning-Kruger Effect — our incompetence masks our ability to recognize our incompetence.

There have been many psychological studies that tell us what we see and what we hear is shaped by our preferences, our wishes, our fears, our desires and so forth. We literally see the world the way we want to see it. But the Dunning-Kruger effect suggests that there is a problem beyond that. Even if you are just the most honest, impartial person that you could be, you would still have a problem — namely, when your knowledge or expertise is imperfect, you really don’t know it. Left to your own devices, you just don’t know it. We’re not very good at knowing what we don’t know

In logical reasoning, in parenting, in management, problem solving, the skills you use to produce the right answer are exactly the same skills you use to evaluate the answer