I'm working on a retrospective pre-post competency survey, and needed to be reminded of some basics of survey design again.
I find Neuman's chapter about survey design a good foundation: http://www.amazon.com/gp/product/0205457932.
Jane Davidson Makes some compelling arguments that suggest that we do need to think twice when we decide to make use of Likert type scales in surveys. http://genuineevaluation.com/breaking-out-of-the-likert-scale-trap/
She sugggests that rather than use the "strongly agree / disagree" type anchors, one could use evaluative terms like "inadequate / good" that might make the data easier to interpret.
Ive also found a list of possible Likert Scale Anchors that are most useful:
http://www.hehd.clemson.edu/prtm/trmcenter/scale.pdf
Thursday, February 10, 2011
Friday, July 02, 2010
Why Competency Self Assessments are essentially flawed beyond redemption
I'm working on an evaluation to determine if a training programme for senior government managers makes a difference - In the competence level of the managers, and in the service delivery they are able to produce within their work context. We are tracking a wide evidence base about all of the participants, but the client is insistent that a competency self-assessment be included. We agreed, on the condition that this one piece of evidence will be used together with all of the other evidence we will be collecting throughout the study. The value that the competency self-assessment will add, is something we have debated in the team. The following entertaining post by Errol Morris, however, pretty much sums it all up:
http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/
David Dunning, a Cornell professor of social psychology... wondered whether it was possible to measure one’s self-assessed level of competence against something a little more objective — say, actual competence. Within weeks, he and his graduate student, Justin Kruger, had organized a program of research. Their paper, “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments,” was published in 1999.
Dunning and Kruger argued in their paper, “When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it. Instead, …they are left with the erroneous impression they are doing just fine.”
It became known as the Dunning-Kruger Effect — our incompetence masks our ability to recognize our incompetence.
There have been many psychological studies that tell us what we see and what we hear is shaped by our preferences, our wishes, our fears, our desires and so forth. We literally see the world the way we want to see it. But the Dunning-Kruger effect suggests that there is a problem beyond that. Even if you are just the most honest, impartial person that you could be, you would still have a problem — namely, when your knowledge or expertise is imperfect, you really don’t know it. Left to your own devices, you just don’t know it. We’re not very good at knowing what we don’t know
In logical reasoning, in parenting, in management, problem solving, the skills you use to produce the right answer are exactly the same skills you use to evaluate the answer
http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/
David Dunning, a Cornell professor of social psychology... wondered whether it was possible to measure one’s self-assessed level of competence against something a little more objective — say, actual competence. Within weeks, he and his graduate student, Justin Kruger, had organized a program of research. Their paper, “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments,” was published in 1999.
Dunning and Kruger argued in their paper, “When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it. Instead, …they are left with the erroneous impression they are doing just fine.”
It became known as the Dunning-Kruger Effect — our incompetence masks our ability to recognize our incompetence.
There have been many psychological studies that tell us what we see and what we hear is shaped by our preferences, our wishes, our fears, our desires and so forth. We literally see the world the way we want to see it. But the Dunning-Kruger effect suggests that there is a problem beyond that. Even if you are just the most honest, impartial person that you could be, you would still have a problem — namely, when your knowledge or expertise is imperfect, you really don’t know it. Left to your own devices, you just don’t know it. We’re not very good at knowing what we don’t know
In logical reasoning, in parenting, in management, problem solving, the skills you use to produce the right answer are exactly the same skills you use to evaluate the answer
Thursday, April 08, 2010
Can a song cause someone's death?
A quantitative evaluator, a qualitative evaluator, and a normal person are waiting for a bus. The normal person suddenly shouts, "Watch out, the bus is out of control and heading right for us! We will surely be killed!"
Without looking up from his newspaper, the quantitative evaluator calmly responds, "That is an awfully strong causal claim you are making. There is anecdotal evidence to suggest that buses can kill people, but the research does not bear this out. People ride buses all the time and they are rarely killed by them. The correlation between riding buses and being killed by them is very nearly zero. I defy you to produce any credible evidence that buses pose a significant danger. It would really be an extraordinary thing if we were killed by a bus. I wouldn't worry."
Dismayed, the normal person starts gesticulating and shouting, "But there is a bus! A particular bus! That bus! And it is heading directly toward some particular people! Us! And I am quite certain that it will hit us, and if it hits us it will undoubtedly kill us!" At this point the qualitative evaluator, who was observing this exchange from a safe distance, interjects, "What exactly do you mean by bus? After all, we all construct our own understanding of that very fluid concept. For some, the bus is a mere machine, for others it is what connects them to their work, their school, the ones they love. I mean, have you ever sat down and really considered the bus-ness of it all? It is quite immense, I assure you. I hope I am not being too forward, but may I be a critical friend for just a moment? I don't think you've really thought this whole bus thing out. It would be a pity to go about pushing the sort of simple linear logic that connects something as conceptually complex as a bus to an outcome as one dimensional as death."
Very dismayed, the normal person runs away screaming, the bus collides with the quantitative and qualitative evaluators, and it kills both instantly...Very, very dismayed, the normal person begins pleading with a bystander, "I told them the bus would kill them. The bus did kill them. I feel awful. To which the bystander replies, "Tut tut, my good man. I am a statistician and I can tell you for a fact that with a sample size of 2 and no proper control group, how could we possibly conclude that it was the bus that did them in?"
* On that note, however, I reject all the talk that the singing of a song CAUSED the death of a man. It may have been a contributor, but I doubt it was either a necessary or sufficient circumstance for this specific death!
http://www.news24.com/SouthAfrica/News/Spike-in-race-tensions-institute-20100406
Without looking up from his newspaper, the quantitative evaluator calmly responds, "That is an awfully strong causal claim you are making. There is anecdotal evidence to suggest that buses can kill people, but the research does not bear this out. People ride buses all the time and they are rarely killed by them. The correlation between riding buses and being killed by them is very nearly zero. I defy you to produce any credible evidence that buses pose a significant danger. It would really be an extraordinary thing if we were killed by a bus. I wouldn't worry."
Dismayed, the normal person starts gesticulating and shouting, "But there is a bus! A particular bus! That bus! And it is heading directly toward some particular people! Us! And I am quite certain that it will hit us, and if it hits us it will undoubtedly kill us!" At this point the qualitative evaluator, who was observing this exchange from a safe distance, interjects, "What exactly do you mean by bus? After all, we all construct our own understanding of that very fluid concept. For some, the bus is a mere machine, for others it is what connects them to their work, their school, the ones they love. I mean, have you ever sat down and really considered the bus-ness of it all? It is quite immense, I assure you. I hope I am not being too forward, but may I be a critical friend for just a moment? I don't think you've really thought this whole bus thing out. It would be a pity to go about pushing the sort of simple linear logic that connects something as conceptually complex as a bus to an outcome as one dimensional as death."
Very dismayed, the normal person runs away screaming, the bus collides with the quantitative and qualitative evaluators, and it kills both instantly...Very, very dismayed, the normal person begins pleading with a bystander, "I told them the bus would kill them. The bus did kill them. I feel awful. To which the bystander replies, "Tut tut, my good man. I am a statistician and I can tell you for a fact that with a sample size of 2 and no proper control group, how could we possibly conclude that it was the bus that did them in?"
* On that note, however, I reject all the talk that the singing of a song CAUSED the death of a man. It may have been a contributor, but I doubt it was either a necessary or sufficient circumstance for this specific death!
http://www.news24.com/SouthAfrica/News/Spike-in-race-tensions-institute-20100406
Thursday, November 26, 2009
A lonely brainstorm... Or many minds?
A grantmaking organization (our client) is interested in evaluating the level of their service delivery and relationship management – as perceived by the grantees that they disburse funds to. So here is the question – What are the evaluation standards that we should use?
Grantee perceptions?
The terms of reference indicates that the client expects that the evaluators will interact with the grantees to answer their questions. But if we ask grantees what they think of the grant maker’s processes, approach, involvement, communication etc. we might get senseless data because the wide range of grantees will have very different expectations about what qualifies as good service delivery / relationship management. It will probably be easy to collect data about their perceptions, but that won’t be very useful. And then there is also the issue of possible bias: Those grantees that experienced difficulty in submitting reports etc for monitoring purposes, might actually be slightly more negative than the rest of the grantees that would probably be eager to be complement the people that will dish out their next pay check.
The Grantmakers’ own standards?
It might make sense to determine whether the grantmaker has any implicit or explicit service delivery standards or contracted agreements that could be used as the standard to evaluate their performance against. But if the grantmaker has a standard that says: “All applications must be acknowledged in writing within 6 months from the date of receipt” that would be easy to check, but surely that service standard seems a little odd? Does it really take six months to respond to a submission?
Industry standards and benchmarks?
The alternative would be to look at service delivery standards and benchmarks as set by other industry players. There’s lots of literature about grantmaking internationally, but information about South African grantmakers are limited – There is the CSI handbook, but it doesn’t contain the level of detail that may be required to develop an extensive set of evaluation standards and benchmarks. And grant makers are notoriously secretive about their approach, systems and quality standards, so we will probably not be able to get detailed information from more than a handful of players in the field that we have established past relationships with.
Room for a participatory agreement on what exactly should be measured?
It is possible that a rigorous engagement of grantees and grant makers at the outset of the evaluation could provide the most satisfactory solution to the “which standards should we use” question. And that is probably just what we will do! Background research about all of the above will probably provide a good basis to start the workshop, but it will be interesting to see what the final consensus will dictate!
Grantee perceptions?
The terms of reference indicates that the client expects that the evaluators will interact with the grantees to answer their questions. But if we ask grantees what they think of the grant maker’s processes, approach, involvement, communication etc. we might get senseless data because the wide range of grantees will have very different expectations about what qualifies as good service delivery / relationship management. It will probably be easy to collect data about their perceptions, but that won’t be very useful. And then there is also the issue of possible bias: Those grantees that experienced difficulty in submitting reports etc for monitoring purposes, might actually be slightly more negative than the rest of the grantees that would probably be eager to be complement the people that will dish out their next pay check.
The Grantmakers’ own standards?
It might make sense to determine whether the grantmaker has any implicit or explicit service delivery standards or contracted agreements that could be used as the standard to evaluate their performance against. But if the grantmaker has a standard that says: “All applications must be acknowledged in writing within 6 months from the date of receipt” that would be easy to check, but surely that service standard seems a little odd? Does it really take six months to respond to a submission?
Industry standards and benchmarks?
The alternative would be to look at service delivery standards and benchmarks as set by other industry players. There’s lots of literature about grantmaking internationally, but information about South African grantmakers are limited – There is the CSI handbook, but it doesn’t contain the level of detail that may be required to develop an extensive set of evaluation standards and benchmarks. And grant makers are notoriously secretive about their approach, systems and quality standards, so we will probably not be able to get detailed information from more than a handful of players in the field that we have established past relationships with.
Room for a participatory agreement on what exactly should be measured?
It is possible that a rigorous engagement of grantees and grant makers at the outset of the evaluation could provide the most satisfactory solution to the “which standards should we use” question. And that is probably just what we will do! Background research about all of the above will probably provide a good basis to start the workshop, but it will be interesting to see what the final consensus will dictate!
Monday, October 26, 2009
Visualization Methods - Really really interesting
Previously I wrote about Edward Tufte's Book on presenting graphs. Well, it seems that data visualization has been taken to a whole new level.
Ralph Lengler & Martin J. Eppler form the Institute of Corporate Communication compiled a "Periodic table" of visualization methods that categorizes and shows examples of about 100 visualization methods.
The table can be downloaded in pdf format at:
http://www.visual-literacy.org/periodic_table/periodic_table_as_pdf.pdf
But try the online version - As you mouse over the various "elements" an example pops up to demonstrate what it looks like.
http://ow.ly/v9RI
The full article explaining the table can be found at
http://ow.ly/wk7d
Wow - It takes people specializing in visualization methods to think of such an innovative way to present their concept.
PS. I heard about this on the American Evaluation Assocation's Linked in Group.This and other useful information gets shared from time to time.
Ralph Lengler & Martin J. Eppler form the Institute of Corporate Communication compiled a "Periodic table" of visualization methods that categorizes and shows examples of about 100 visualization methods.
The table can be downloaded in pdf format at:
http://www.visual-literacy.org/periodic_table/periodic_table_as_pdf.pdf
But try the online version - As you mouse over the various "elements" an example pops up to demonstrate what it looks like.
http://ow.ly/v9RI
The full article explaining the table can be found at
http://ow.ly/wk7d
Wow - It takes people specializing in visualization methods to think of such an innovative way to present their concept.
PS. I heard about this on the American Evaluation Assocation's Linked in Group.This and other useful information gets shared from time to time.
Subscribe to:
Posts (Atom)